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tr; to define all tbe competencies required collectivelj' by all tbe 
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conclusions, draun froa tbe results) is also described. (lutbor/lltV) 
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INTRODUCTION 

One of the "close down products** of the Ohio State Untverst ty-based 
Mode) Training Project requested by NIE personnel was a wrap-up of the 
current phase of development on the Self Assessment of Evaluation Skills 
(SAES) instrument, with documentation and reporting of al) work completed. 
That documentation and reporting are the purpose of the report which 
fol loviS* 

Numerous persons have been involved with SAES since its inception, 
and their contributions to the work reported below are acknowledged at 
this time. Dr. Mary Anne Bunda and Dr. Daniel Stufflebeam should be 
recognized for the initial conceptualization of the SAES study, for the 
development of the SAES instrument, and for continued interest in the 
SAES project. Students from the Model Training Project who have been 
actively involved in the SAES study are Unhai Ahn» Phyllis Fa1k» and 
Andrea Lash. Specifically, the correlational analysis of the SAES data 
reported below was completed by Ahn; the follow-up of nonrespondents to the 
SAES instrument was conducted by Falk; and the^item contamination study 
was completed by Lash, Nancy Riddle* Sharon Kelley* and Jennifer Hixson 
have been able secretaries to the SAES staff, and their help throughout 
this study is appreciated* 
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BACKGROUND OF THE STUDY 

The Model Training Project (MTP) based at The Ohio State Uni vers ! ty 
had as its ffiajor purpose the development of a trainin9 program which 
would^ when implemented^ meet the total range of evaluation training needs 
within the member agencies of the MTP consortium. (Readers are referred 
to the document by Hllderbrand, et al . ^ in this final report series which 
describes that consortium.) If the MTP's goal was achieved^ staff members 
fe)t that this newly-developed program would provide one model of the 
types of evaluation training needed throughout the field of education* 

During early Winter, I97l> the HTP staff held a retreat to concen- 
trate on project development. A concern felt by most staff members was 
that training in evaluation should not become ^ set curriculum for all 
students. Rather, the staff members felt that some process which would 
allow individual program planning would better fit the structure of 
the HTP. 

One o^ the major premises of the MTP was that evaluation is a team 
activity, and therefore persons must be trained to fill different positions 
on'sucK^a team.^ It follows, then, that persons must be trained differently, 
depending on their positions on a team — i.e., depending on their evalua- 
tion role.' 

A problem exists, however, in that evaluation roles are presently at 
a primitive stage of definition. The skills and knowledge needed by persons 
who perform different evaluation roles are not widely recognized; even 
among those persons writing in this area, there Is no consensus of just 
what it takes to be a ''competent evaluator.** Therefore, one of the first 
jobs for members of the MTP staff was to try to define all the competencies 
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required coUectively by all the members of an evaluation team. That 
defTnitionof ^kills and knowledge is called the UnTverse of Evaluation 
Competencies (UEC) . Mainly the result of a conceptual effort^ it was 
recognized that the UEC must be validated by empirical studies. The 
Self Assessment uf Evaluation Skills (SAES) instrument was developed for 
that validation purpose#>^ 

The UEC was divided into ten major sections^ each of which represented^ 
in the view of the authors cf the UEC, a necessary component in the 
buildtng of an evaluation profession. These ten sections were: 
1} Administrative Leadership in Evaluation^ 2) Research in Evaluation^ 
3) Development of Evaluation Methodology, k) Instruction in Evaluation, 
S) Implementation in Evaluation, 6) Development of Evaluation Systems, 
7) Diffusion of Evaluation Developments, 8) Use of Evaluation in Educa- 
tional Leadership, 9) Use of Evaluation in Teaching, and 10) Use of 
Evaluation in Curriculum Development. Skill areas were defined for the 
ten major sections; further, competencies were specified for each of the 
skill areas. It is from this third level of the UEC that the SAES 
instrument was developed. 

^The UEC was checked for completeness against pertinent past research 
(e.g., the AERA Task Force on Research Training data, the Oregon Studies 
data). Each time a competency was found in that research which was not 
Included in the UEC, it was added to the UEC. In addition, one of the 
researchers in this field of investigation (Blaine Worthen, the director 
of the AERA Task Force) examined the UEC to determine its comprehensiveness. 
After these two content validation steps, MTP staff felt relatively com- 
fortable about the rnclusiveness of the set of competencies listed In the 



UEC^ and also felt that the UEC and the related SAES Instrument could serve 
as a basis for empirical research and program development. 

Vhree uses were to be made of the SAES instrument. First, and most 
important for the purpose of validating the UEC, a national survey of 
members of the evaluation profession was to be conducted. Second, students 
in an evaluation training program could check their level of competence 
in certain skill areas with SAES. Third, learning experiences (e.g., 
classes, simulations) could be described on jthe basis of which skills 
from SAES they were designed to transmit. 

When the instrument was first written, it contained approximately 300 
Items. The items were grouped according to the ten major sections of the 
-UEC. Respondents (MTP straff and some students) were asked to rate their 
level of competence in each skill on a scale from I (no competence) to 
5 (expert in the field). 

The first revision of the instrument included a formal change and 
clarification of ambiguous items.' The tittes of the ten UEC sections were 
dropped to lessen the possibility of bias in responses. The instrument was 
divided into three content equivalent forms of 113 items each (with some 
overlap of items among forms). This was done for purposes of practicality; 
ft was hoped that shortening the instrument would Increase the response 
rate tn the national survey of practicing evaluators mentioned earlier. 
Also at this time, separate directions and answer sheets were prepared, 
corresponding to the three planned uses of SAES, for 1) practicing evalua* 
tors, 2) students, and 3) learning experiences. 

After this revision, two groups of persons responded to the instrument 
in relation to their ievet of competence for each of the skills listed in 
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all three forms of the Instrument. The two groups were I). all students 

in an evaluation class at Ohio State University* and other MTP students 

not in the class, and 2) leaders In the field of evaluation who comprise 

the Working Group on Evaluation. A Q-f actor analysis was performed on 

the data to see ir the instrument would differentiate between students 

and practicing evaluators. The results of that analysis showed* In fact» 

perfect discrimination. Specifically, the first two factors differentiated 

the two gi'oups of individuals perfectly with eigen values of 5.85 and 2.^» 

respectively. The first factor accounted for 20% of the variance and the 

second factor accounted for an additional 9% In an orthogonal space. 

Also at this time» the three forms of the SAES instrument were 

analyzed to determine the equivalency of scores a^ong the three forms 

and the internal cons i stency wi thi n the three forms. The results of 

that analysis were as follows: 

Form lA mean response - 1. 96 (on a scale from 1*5) 
Split-half reliability - .99 

Form 'B mean response - I. 96 (on a scale from 1-5) 
Split-half reliability - 1.00 

Form IC mean response - 1.87 (on a scale from 1-5) 
Split-half reliability - .99 

^In July» l972» representatives from the MTP consortium agencies 

critiqued the UEC and SAES. Several concerns about SAES were raised* 

prompting the MTP staff to engage consultants to work with the SAES staff 

in further revision of the instrument. The SAES instrument whicM resulted 

from these revisions was a 23^-itera instrument which was to be used by 

practicing evaluators in answering three questions: I) How competent are 

you in this skill? 2) How important is this skill to the successful 

completion of your duties? and 3) How interested are you in performing 
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this skill?^ 

The two new response scales* importance and interestt were added 
because the MTP staff and consortium representatives were hesitant to say 
that just because evaluators presently practicing in the field have certain 
competencies in certain skills, a training program should be developed on 
the basis of that data alone.' The two new scales were developed to allow 
more inforffiatton to be available to the HTP staff and other curriculum 
developers . 

^This version of SAES was administered to a small group of students 
and incumbent evaluators and was found to be relatively free of ambiguities. , 
The instrument was judged* then* to be ready ror pilot testing, tn the 
remainder of this report* that pilot test (including methods and procedures 
used* results obtained* and conclusions drawn from the results) is described. 
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PROCEDURES 



The original purpose of the pilot test was to test the usefulness 
{ev9;f applicability and understandabJ Hty) cff the SAfS tnstrument on a 
small group of practicing evaluators before the full national survey 
was undertaken. Also* the SAES staff hoped to learn two additional 
things: 1} would the data received front the ptiot test be amenable to 
the analyses that were planned for the full survey* and 2) what percent 
tage of returned questionnaires might reasonably be expected for the 
full survey. In order to answer those questions* several different 
methods and procedures were used. A descrfption of the procedures is 
included in this section of the report. 

Gen eral Procedures 

In this section* general procedures relating to the distribution of 
the SAES instrument are described. Included are the population and sample 
for the pilot test* questionnaires and cover letter used in the pilot 
test, data collection* and statistical treatment of the data. Following 
this section* descriptions are given of additional substudtes whi^ch^^were 
undertaken in conjunction with the pilot test in order to answer specify 
tcally certain questions noted above. 

Population and Sample . The population of concern jn this ^tudy was 
an incumbent educational evaluators. It was uncertain how much of this 
experimental population was accessible. Four source^ were used to 
complete a directory of practicing evaluators; approximately 2*5^0 persons 
were identified through these sources. The SAES staff believed that 
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these na(nes represented a large enough proportion of the experimental 
population that the listing of names could» for all practical purposes^ 
be considered complete. 

7he four sources which were used to identify Incumbent evaluators 
ar^ I isted belcw. , 

1) The meihbetship roster from Otvislon H (School Evaluation 

and Program Development) of the American Educational Research 
Association 

2) The mailing list for the interf acer » a newsletter published 
by the Model Training Project 

3) Two lists of evaluation consultants compiled by Robert Stake 
^) Personal address files of several evaluation specialists 

tn the full survey^ it was planned that all persons in the identified 
population would be sent a SAES instrument, tn the pilot test» however^ 
a 10% sample was deemed sufficient to answer the questions of Interest. 
Accordingly^ a stratified random sample of 252 persons was drawn» with 
address (either home or office) being the stratifying variable. Of the 
252 persons in the sample^ 132 had office addresses and 120 had home 
addresses; this was the proportion of office and home addresses found in 
the ejtoerimental population. 

(Questionnaires and Cover Letter . As previously discussed^ the SAES 
instrument was already developed and ready for use (a copy may be found 
in Appendix A of this report). Other materials which had been prepared 
previously and r^'^ded only minor revisions to be sent with the instrument 
were the Oirections for Completing the Self Assessment of Evaluation 
Skills Instrument^ the Answer Sheets the Item Critique Torm^ and the 
Role Description Questionnaire (a copy of each is included in Appendix B). 
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The Role Description QMestlonnai re contained items Intended to gather 
persona) ogtca) data about questtOnnaIre respondents. Qjuestions centered 
on the respondent's educational background and his eniployment respOn<;{* 
bttfttes. One purpose of the pilot test was to validate the Zk role 
descriptions listed in the Role Oescrlptfon QjUestlonnai re. It was hoped 
that tn the full survey a checklist-type questionnaire could be developed 
to gather this personalogical data. 

The cover letter accompanying the SAES and other instruments was a 
multillthed letter with each persOn^s name and address typed on the letter. 
The letterhead of the stationery read "Comi ssion for the Study of the 
Evaluation Profession." This Cocnmlsslon was organized by the SAES staff 
for two purposes: 1) to provide on-goIng critiques of the instrufnents^ 
procedures^ etc.^ to be used In both the pilot test and the full survey^ 
and 2) to provide credibility for the study through the Individuals listed 
as members of the Cofmission. Readers are referred to the Bunda paper 
(which appears later In this report) for a more conplete description of 
the work of the Conmlssion. 

Data Collection . The initial mafMng of cover letter, SAES, and other 
materials was made on November 10, 1972. Three weeks later, on Noven^er 
30, )372, a follow-up reminder was sent. As wilt be explained In ? later 
section of this report, half of the non respondents at this time received 
a postcard follow-up and the other half of^he nonrespondents received 
a foDow-up letter with alt the instruments and enclosures that had been 
included in the initial mailing. A second foMow-up reminder, this time 
a postcard for all nonrespondents, was sent on December 15* 1972. 
January 12, t973> was the date chosen as the cut*off, beyond which returned 
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questionnaires were not Included in the analyses. 

Each packet of materials was nuihbered so that returned questionnaires 

could be checked off a master list. As wt 1 1 be discussed later^ one of 
three types of return envelope was included tn each packet^ the three 
types of envelope being Gtcirq^ed^ business repty^ and unstanq^ed. (Copies 
of all folloW-up correspondence may be found in Appendix C.) 

Three Substudies 

In the next three Sections^ specific procedures are described relating 
to three substudies undertaken to answer these questions: 

1) How could the response rate be increased without Increasing 
the cost of the study? 

2) Are the SAES Items clear? 

3) Why did sotne persons fail to respond to the request to 
complete the SAES Instrument? 

Procedures for Mailing Costs Substudv * As in any mailed question- 
naire study^ the SAES staff was concerned about maximizing response rates* 
In most cases^ the validity of the results of a questionnaire study Is 
only as good as the response rate^ I.e*^ the higher the response rate^ 
the more confidence one can place in the results. There are many tech- 
niques which researchers employ to try to attain high response rates^ 
and some of the techniques (such as personal or telephone follow-up) do 
seem to be effective in increasing return rates^ 

Unfortunately, some of the more effective techniques are quite 
expensive. In working with a pilot test sai^jle of 252> the SAES staff 
could have felt relatively free to do whatever they could to achieve high 
response rates, regardless of the cost. Since a primary purpose of the 
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pilot test was to estimate the response rate the full survey might achieve 
howevert the SAES staff could do nothing in the pilot test that they would 
not do in the full survey. With the nunber of potential respondents set 
at 2^500 in the fuil survey^ cost did become a factor. In the pilot test» 
therefore^ two variables were investigated to determine the most cost- 
effective means for achieving a high response rate. 

The two variables Included In this substudy were mode of follow-up 
and format of return envelope. A completely crossed 2x3 factorial 
design (two levels of fo(low-up» three levels of return envelope) was 
employ^. Pilot test respondents were randomly assigned the six cel?s» 
with the constraint that the population proportion of respondents with , 
home or office addresses was maintained in these six cells. (In actuality 
there were 3^ cells In the design. A third classification variable 
relating to the item critique substudy was included In the experimental 
design;'there were six levels of that variable.) 

The two levels of follow*up were postcard and letter (with SAES and 
all other materials from the Initial mailing included for a second time). 
This dichotomy was studied for the first follow-up contact onTy; at the 
time of the second follow-up^ all nonrespondents received a postcard 
reminder. 

The three return .envelope formats were stamped* business reply^ and 
unstamped. Except for the postage* all envelopes were the same. The 
effect of this variable was studied through the first follow-up as well. 

Procedures for Item Critique and Item Contamination Substudies . 
Persons who had critiqued the SAES instrument questioned the appropriate- 
ness of items containing two verbs in a single item. These Items forced 
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respondents to produce one rating for what might be two separate skHts. 

In order to determine if information on respondents' skills was lost 
in the double verb items (because respondents may have to average the 
ratings for each skill or respond to one skill only)* a substudy was 
included in the SAES pilot test. The purpose of the substudy was to 
see if responses to double verb items (implying two skills)^ differed from 
responses to each. of two single verb items measuring the same skills. 

All double verb items were noted in the SAES instrument and reviewed 
by the SAES staff. Items whose verbs could imply two separate skills 
(e.g.t^ to design and administer an evaluation)* or two levels of one skillt 
were chosen for the study. Twenty-eight such items were chosen for 
inclusion. 

Each double verb item was then rewritten into two separate items (so 
respondents could answer each item separately). These new^ single verb 
items were reviewed by the SAES staff for clarity and appropriateness* 

Samples for three cells were randomly drawn from the populaxion* 
which at this time contained the approximately 2*500 names minus the 252 
persons in the pilot test sample. A total of 120 persons was drawn and 
^0 persons were randomly assigned to each cell. The population's propor* 
tfon of home and office addresses was maintained in the sample and in 
the eel Is. 

Each respondent received the Role Description duestionnaire* a stamped 

return envelope* and a set of SAES items. The packets were mailed 

November 17* 1972. 

Group I received all new single verb items (N « 5^) and ti single 
verb items randomly chosen from the original SAES for a total 
of 6^ items. 
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Group 2 received 1/2 of the new single verb items {N « 28* one item 
randomly chosen from each pair of Items) and the same 8 single 
verb items from the original SAES for a total of 36 items. 

Group 3 received alt 28 double verb Items and the same ti single 
verb items from the original SAES for a total of 3^ items. 

The eight single verb items chosen from the original SAES were 
included so responses to these Items could be compared among the three 
groups. It was assumed that mean responses to these items would be 
(statistically) the same for all three groups. 

Subjects who had not returned their answer sheets were sent follow- 
up reminders on December 7f 1972. These follow-ups included the same 
material that was in the original mailing. (The Items included in this 
substudyt and all materials sent to the respondents In this sample* are 
included In Appendix 0.) 

In addition to this substudy* each person In the pilot test sample 
of 252 critiqued a portion of the SAES Items or the Role Description 
Questionnaire. The total sample of 2^2 persons was divided Into six 
groups: five groups critiqued 5O Items each (for clarity and ease In 
responding)* and one group critiqued the Role Description Questionnaire. 
For the 23k SAES Items* respondents answered two Likert-type Items for 



each SAES 


i tem. 


The 


descriptors for the ciarity scale were: 


0 


The 


i tem 


was completely understandable. 


2) 


The 


Item 


was moderately clear. 


3) 


The 


i tem 


was moderately ambiguous. 


^) 


The 


1 tem 


was completely ambiguous^ 


The descri 


ptors 


for the ease of answering scale were: 


I) 


The 


i tem 


was easy to answer. 


2) 


The 


i tem 


was moderately easy. ' 
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3) The Item was moderately difficult. 
^) The item was difficult to answer, 
the critique of the Role Oescriptfon Quest lonnat re was conducted by 
answering five open-ended Items about the questionnaire. 

Procedures for NonresPQndent Bias Check . In order to assess possible 
differences between respondents and nonrespondentsT^ a telephone follow-up 
of nohrespondents was conducted. Approclmately fifteen percent of the 
nonrespondents (N ^ 22) was randomly selected to be included In the foMow-up 
sttitly. This fifteen percent of nonrespondents was compared wfth approc- 
tmately fifteen percent of the respondents who returned usable question* 
na { res . 

In the pilot study/ the sample had been dWIded into several subgroups. 
Sofiie of the sample fiad borne addresses while soote had office addresses. In 
each of these two groups^ the sample was further divtded by types of 
return envelope: some persons received stamped return envelopes^ some 
received business reply envelopes^ and some received envelopes with no 
return postage. A proportionate number of respondents and nonrespondents 
were sampled from each of the subgroups. The number of members in the 
subgroups was often not conducive to proportionate sampling within the 
restrictions of a fifteen percent sample. For cases in which an addi- 
tional member was needed to complete the fifteen percent sample^ the 
subgroup from which the additional member was drawn was randomly selected. 

The respondents and nonrespondents were compared on the following six 
variables: 

1) position title 

2) role descri[>tIon 

3) employing institution/department 

18 



^) length of time in present position 

5) highest degree 

6) date of highest degree 

In addition to the six variables listed above* another variable 
was investigated. )n order to obtain an Indication of the reasons for 
their failure to respond to the questlonnal re* the nonrespondents were 
queried In regard to this Issue* 

Ih the next section* general results as well as results for each of 
the substudles described above are presented* The statistical treatment' 
of the data Is discussed as an Introductory segment to each set of 
results* 
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RESULTS 

tn this section of the report^ results from the various substudies 
and other analyses are given. In addition^ the stattsttcal techniques 
uted in each analysts are listed before the results are described. 

General Results 

Thirty-one percent (N * 77) of the 252 SAES Instruments malted to 
respondents were returned to the SAE? staff with completely usable^, 
responses. An additional eight questionnaires were partially completed 
and returned^ while 21 were either undeUverable or returned blank. 

The Rote Description Questionnaires returned by the 77 persons who 
responded comptetety to the SAES instrument provided information about 
the educational and professional experience of the respondents. Table ) 
shows the distribution of respondents among: the various categories 
contained In the quest lonnalre. Professional experience data relates 
to the respondents' present position. Because respondents could choose 
more than one descriptor in certain categories (e.g.* university admini- 
strator and university professor)^ the figures below do not always sum 
to 77. 

Table I 

Description or Respondents 
to SAES Instrument (N * 77) 

Present Job Title 

Administrator: School building - I Administrator: College or univ. - 1^ 

Teacher: Elementary or secondary •* I Professor: College or univ. •* 26 
RDE Spec^: School building - 1 RDE Spec: College or univ. - 5 

Administrator: School district - 6 Administrator: R&D Lab - 2 

RDE Spec: School district * 5 Specialist: R & D Lab - ^ 

(Table I continued) 

i 

Research^ Development^ or Evaluation Specialist 

20 

16 
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Table ) (continued) 

Descrtptton of Respondents 
to SAES Instrument (N * 77) 



Present Job Tttle (continued) 

Specialist^: School district - I Other administrator - 8 

Administrator; State dept. of edbc. - I Other ROE specialist 3 
RDE Spec: State dept. of educ. - 4 



Present. Role Description 



Administrator; Research 11 
Administratoi ; Evaluation - 9 
Administrator: Development ^ I 
bther-admihistrjatbr - 19 
Professor: Research courses - 9 
Professor: Evaluat ion courses 3 
Prqfessbr: Development courses * 6 
Other college professor * 16 
Teacher; Elementary or secondary - 1 
Specialist: Evaluation design II 
Specialist: Evaluation implementation* 
Consultant; Evaluation 10 
Educational program auditor 2 



Other consultant - 2 
Product developer - 2 
Program developer - Z 
Researcher - 

^StiUstician/^^ analyst - 4 
Psychologist • T 
Specialist: Dif fusion/dissemi-^ 

nation - 3 
Evaluator of federal projects - 8 
Counselor 2 
7 Director of testing program - 2 
Institutional researcher - 2 
Other - I 



Present employing 

University/Education - 28 
University/Psychology - I 
Univ. /special ly funded project 

or center - 8 
School system/elem. school - I 
School system/sec. school - I 
School system/eval uat ton» account- 

ability^ and/or planning and 

research - 8 
School system/special education ^ I 



Agency/Oepartment 

School system/test {ng» student 
information - I 

School system/guidance^ pupil per- 
sonnel seVvices 2 

Lab or Center/specially funded 
projects - 6 

'State dept. of educ./research^ evaT 
uation^ planning - 3 

State dept. of educ. /special 
projects - 2 

Other - 9 







Length of Time in Present Position 




Less than 1 


year 


-13 3-5 years - 20 


More than iO years - k 


1 - 2 years 


' 22 


6 * 10 years - 8 








Degree Required for Present Position 




Bachelor 's 


- 3 


Master's ^ 21 Master's plus - 2 


Doctorate - 39 



(Table I continued) 



Specialist at district level, e.g. , ^ psychologist 

21 
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Table t (continued) 

Description o^ Respondents 
to SAES Instrument (N ^ 77) 

Certification Required for Present Position 

None - 4^ Supervision - 2 Social Work t 1 

Teaching - 5 Administration - k Other * k 



It would appear from looking at Table t that the sample was over-* 
represented with persons from colleges and universities. This is under- 
stahdablet since it is likely that the vast majority of 7VERA members 
(the primary source of names in the SAES staff's directory of evaluators) 
are college or university personnel. Before the full survey could have 
been undertakent it would probably have been necessary to augment the 
directory with the names of (possibly numerous) evaluators from settings 
other than universities. (This point is discussed further in the 
"Limitations of this Study" section later in this report.) 

Results of Correlation Analysis 

The 23^ SAES items which were sent to the pilot test sample were 
developed from the Universe of Evaluation Competencies* which has been 
described in an earlier section of this report. The 23^ items were 
grouped on an a priori basis into eight categories of items; each 
category was thought to represent a distinct skill area necessary for 
the successful completion of evaluation activities. One of the purposes 
of the full survey was to test empirically this logical grouping of 
items through the use of factor analytic techniques. Because of the 
small number of responses to the pilot test (N « 77)* it was impossible 
to factor analyze the pilot test results* but the SAES staff wanted to 
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have some idea (even though it might be tenuous) of the validity of the 
eight logical groups. For this reason, correlation analyses of the 
pilot test data were performed. 

Two different correlation analyses were performed. One was under* 
taken to determine inter-item correlations among the 23** items. The 
BHD X8^ program was used for this analysis. The second analysis was 
concerned with inter-scale correlations (i.e., the tntercorretations 
amonc; the competence, importance, and interest scales). The BMO OJD 
prog-am was used for this analysis. - 

Inter-item correlations . The eight logically derived areas or 
categories of skills comprising the SAES instrument are as follows: 
t) knowledge of innovation in evaluation, 2) public relations, 3) data 
processing, k) educational measurement, 5) evaluation administration* 
6) relating evaluation to relevant disciplines, 7) communications, and 
8) research design analysis. A list of the items corresponding to 
each skill area may be found in Appendix E. 

for this analysis, items were grouped according to the category in 
which they were placed on ^ priori grounds. An Inter-item correiation 
matrix was computed for each of the eight categories of items. Then the 
items were put back in numerical order (i.e., they were no longer grouped 
according to the eight categories) and another inter-item correlation 
matrix was formed. Using a table of critical values for the correlation 
coefficient (Glass & Stanley, Statistical Methods in Education and 
Ps vcholoqv ^ p. 536), all correlations which were significant at the .01 
level of confidence were roted. Table 2 shows the percentage of signifi- 
cant correlations in each category as well as the percentage of signifi- 
cant correlations in the total SAES instrument. That is, the percent 
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of significant correlations between an item and all othf^.r tt«ms n o t 
in Its logical group is the number shown in the ^Total" entry. 

Table 2 

Percent of Significant Inter-item Correlations 







Number of 


Percent of 






Items in 


Signi f leant 




Category 


CateQOrV 


CorreUt Ions 


1. 


Knowledge of {nnovation in evaluation 




72.8 ' 


2. 


Publ ic relat ions 


8 


66.7 


3. 


Data processing 


n 


87.2 




Educational measurement 




65. k 


5. 


Evaluation administration 


50 




6. 


Relating evaluation to relevant disci- 








pi ines 


12 


84.8 


7. 


Communications 


22 


Wf.5 


8. 


Research design analysis 


51 


63.2 




Total 


23k 


40.2 



As shown in Table 2» items within each category appear to have 
higher i nterccrrelat ions than the items between categories. The cohesive- 
ness of the items was most evident in categories I* 3t St 6* and 8. 
These five categories had more significant correlations (p < .01) and 
categories 2 and ^ had more significant correlations (p < .05) than the 
intercorrelations among tSe total set of items (the ^fO.2% figure noted 
above). The percent of correlations in category 7 was not significantly 
different from the total group. 

Intftr-scale correlations . Because it was important to determine the 
independence of the three response scales (competencet interest* and 
importance)* an inter^scale correlation matrix was computed for the total 
set of items and for the eight logically derived categories of items. 
The results of that analyf^is are shown in Table 3* 
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Table 3 

Average lnter*$cale Correlations 





Cdt6qory 


Nwtber of 
Items In 
Cateciory 


CI 


cm 


tm 


1. 


KrKMledge of tnnovatton in 












evaluation 




.55 


M 




2. 


Public relations 


8 


.5? 


.33 




3. 


Data processing 


11 


.5*f 


.35 






Educational measurement 




.53, 




.60 


5. 


evaluation administration 


50 


.56 


.52 


.60 


6. 


Relating evaluation to 












relevant disciplines 


12 


.56 


M 


!62 


7. 


CofTvnunl cat Ions 


22 


.52 






8. 


Research design analysis 


5> 


.55 




.59 




Total 


2^k 


.5U 







NOTE; c « competence^ I « Interest^ m Importance 



The difference between and «6l is significant at the «05 level, 
but becauSB of the number of correlations computed, this might be 
expected by chance* 

In general, there were high correlations among the three scales* 
The correlation betv^een the Interest and Importance scales (r[„^) was the 
highest, with an average of *$9« The correlation between the competence 
and interest scales (^c|) second hlg£*.ast, *S^* The correlation 

bctv^een the competence and Importance scales i^^f^) was the lowest* .^* 
However, there was no significant difference among these three correla* 
tions. This overall trend was also evident when Items were grouped for 
each category, with the exception of categories 2 and 3* 

It is difficult to draw any clear conclusions from the above results* 
tt seems safe to say, however, that the three scales are not really* 
independent. Before the full survey would be conducted. It would be- 
wise to determine if si I three scales are actually needed* 
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In Appendix r» inter-scale correlations for all items may be found. 

Results of Item and ttote Oescription Qgestionnaire Critique 

As previously discussed^ the 252 persons in the oilot test sample 
were randomly assigned to six groups for the purpose of critiqueing the 
23^ SAES Items and the Role Description Questionnaire. Responses within 
these six groups were as follows: 



Group Responses Percent 

1 (critique of items 1-50) II 26.2 

2 (critique of items '5V-100) \k 33.3 

3 (critique of items 101*150) 9 21.4* 
k (critique of Items 151*200) 17 ^i0.5* 

5 (critique of Items 201-23^) 9 2U^ 

6 (critique of Role Description Questionnaire) II 26.2 



'V The difference between 21.^ and ^.5% significant at the .05 
level. That should not be a limitation^ hot^ever^ of the results 
reported in this section. 

The analysis procedures for the item critique included two main 

tasks: I) computing descriptive statistics (mean» standard deviation^ 

titandard error* range) for each item by using the BHD 010 programt anti 

2) identifying items which might have problems of ambiguity. This second 

analysis was performed by surrounding the mean of each item In both the 

clarity and ease of answering scales (see pp. 13*14) with a confidence 

interval. If X + t<r- > 2.5 for an item on either of the two scales* the 

Item was determined to be possibly ambiguous and in need of a further 

check. This further check was performed by referring to a notebook which 

had beer kept on all SAES Items. The notebook contained 23^ pages^ one 

for each item. Anytime a person provided a written comment about an Item 

in addition to his numerical rating of the it&n» the comment was recorded 

in the notebook. 
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The mean and standard deviation for each item on the competence 
scale Only are provided in Appendix P. In Appendix G» the items which were 
revised as a result of the item critique appear. 

The Role Description duestionnaire critiaue was performed by answering 
five open*ended questions. Host respondents agreed that most of the ques* 
t lOns Were conducive to the checklist format. The role descriptions 
were not particularly useful ; 8^f the 11 respondents in .this group had 
trouble identifying themselves according to the role descriptions. A 
point made by several respondents was that they performed several of the 
tasksj but for less than full time or even for less than half time. For 
this reason^ it was planned that in the full survey^ a list of roles 
would be provided^ beside which a series of columns would be placed. Each 
column would represent a different amount of a person's professional 
t5me (e.g.» 0-20%^ 21-^*0%). In that format^ if a person usually spent 
One day per week consulting on evaluation problems^ he would simply 
place a check in the 0-20% column opposite the "Consultantievaluation" 
entry. 

Results of Item Contamination Study 

For the competence scale only^ the mean response to each double verb 
item was compared to the mean response of the two single verb items formed 
from the double verb item. For example^ item 1 on the double verb list 
was divided into item I7 (on the list of 36 single verbs and on the list 
of 6^* single verbs) and item 57 (on the list of 6^ single verbs). A 
one-way analysis of variance was performed comparing the mean response of 
items It I7t and 57 on the competence scale. The purpose of having two 
lists of single verbs (one with 36 items and one with 64 items) was to 
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determine if the length of the instrument had anything to do with response 
rate. An analysis of variance was performed on the response rates for 
alt three instruments (36 single verbs^ 6^ single verbs^ 36 double verbs) 
and no significant difference was found. 

Of the 36 double verb items^ k had means which were significantly 
different when the items were divided into two single verb items (2 at 
the .05 levels 2 at the .025 level). The means from the eight single 
verb items from the original SAES were not significantly different among 
the three groups. From this^ one coutd imply that the three groups were 
quite well matched and the four significant differences were^ in fact^ 
real differences. Since the number of items which might have been 
expected to differ by chance alone is fewer than two at the .05 levels 
the fact that four items were significantly different does seem to 
corroborate the belief that some information might have been lost by 
including two verbs in one item. It ts likely^ therefore^ that in the 
SAES instrument to be used in the full survey^ the double verb items would 
have been separated tb for^ single verb items. 

Results of Nonrespondent Bias Check 

The purpose of this substudy was to see if respondents differed from 
nonrespondents on ary variables of interest. If the two groups did differ^ 
the results of any analyses performed on the responses would likely be 
biased to some unknown. degree. If the groups did not differ^ it was 
possible that the results could sttll be biased; but the likelihood of 
that occurring was considered to be less than if the groups did differ. 

Chi*square tests were performed on the six variables of interest 
(present position title, role description, employer, length of time in 
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present positiortt highest degree* date of highest degree)* No significant 
differences in any of these variables were found between the 15% randomly 
sampled nonrespondents and the 15% randomly sampled respondents* (The 
answers to these questions were obtained from the respondents by looking 
at their Role Description Questionnaire*) Thus» it may be said that at 
least as these variables might have affected responses to SAES items» 
there was little evidence of a nonrespondent bias* 

In answer to the question of why the nonrespondents failed to return 
*the SAES instrumentt a total of stx reasons were given* Those reasons 
(and the number of persons giving that reason) are shown in Table ^* 
Some persons gave more than one reason* 

Table k 

Reasons for Nonresponse to SAES 



Number of Persons 

Reason GivinQ this Reason 



Changing jobs 2 

Length of the instrument 11 

Not an eval'uator 6 

Sent in (but never received by SAES staff) k 

Too busy with other responsibilities I 

Hot forwarded from previous job 2 



The length of the Instrument^ it appears* was a very definite factor 
in the low response rate. Because ot this* a decision was made by the 
SAES staff that item sampling procedures would be used in the full survey* 
A more detailed discussion of this topic will be given in the '^Directions 
for Future Research" section of this report* 
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Results of Maninq Costs SubstudV 

The purpose of this substudy^ It will be remembered » was to determine 
the most cost-effective means for Improving response rates. Two forms 
of fol low-up (postcard and letter) and three formats of return envelope 
(stampedV business reply» and unstamped) were used In the substudy. 

Analyses of variance on the return rates of each group in the 2x3 
factorial design were computed using the BHD 08V program. Separate 
analyses were perfonned on the home and office address groups. Analyses 
were performed for two time periods: t) returns as of l2/l(i/72 (after 
one fot1ow*up)» and 2) returns as of 1/12/73 (after two follow-ups)* 
For these analyses only . Incomplete and undel I verabte responses were 
counted as full responses. 

After one follow-up^ there were no significant differences in response 
rates among the home address group in either form of foUow-upt format 
of return envelope^ or the interaction of these two variables. Within 
the office address groups there was no difference among forms of follow- 
up or in the i ntefactiton^term^ but there was a difference (p < *05) 
among format of return envelope. A Tukey test showed that the business 
reply envelope had been returned significantly more often (p ^ .05) than 
the unstamped return envelope. 

After the second follow-up (which^ it will be remembered^ was a 
postcard'tD all nonrespondents)^ there were again no differences among 
response rates for the home address group. The same pattern of significant 
differences occurred for the office address group as In the first analysis. 
This titne^ however^ the difference among response rates for the three 
return envelopes was significant at the .005 level. Tukey tests showed 
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the business reply envelope to have been returned more often than the 
unst^tmped envelope (p K .025) and the stamped envelope to have been 
returned more often than the unstamped envelope {p <#0I). 

These results prompted the following decisions concerning the full 
survey to be made by the SAES staff* First* the much less expensive 
postcard follow-up would be used* Second* the rrost economical return 
envelope appeared to be the business reply (since payment for postage 
was made only on those questionnaires which were returned)* and so they 
would be included in the full survey* 
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CONCLUSION 

The Self Assessment of Evaluatlo^t Skills (SAES) described In this 
report^ including the Instrument's development and testing^ would have 
been a necessary precursor to any national survey of evaluators in which 
the instrument was used. Similarly^ instrument development activities 

such as those described here would be required before any kind of program 

It 

planning could be carried out using the SAES instrument as a basis. The 

^^^^^^^^^ 

requisite first steps have'been taken in the development of an instrument 
designed to I) describe the members of the educational evaluation 
profession^ 2) guide students in choosing appropriate courses of study 
in the evaluation fletd> and 3) alert program developers to areas where 
additional training Is needed so they may Incorporate those areas into 
evaluation curricula. Unfortunately^ it appears at this time that those 
first steps are all that will be taken. 

Pilot test results are by nature of limited general Izable value^ 
for the intent and purpose Is to precede and provide information for a 
larger study to follow. Therefore^ these pilot test results are of 
limited value beyond the study for which the pilot test was to be a 
part. In addition to this major caution^ certain other cautions should 
be made to persons who might read these results and wish to draw Inter* 
pretations from them. These cautions are discussed as "Limitations of 
this Study." Similarly^ some persons reading this report may see atten- 
dant areas they might like to study> and for those persons the "Directions 
for Future Research" section is added. 
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Liihitations of this Study 

The most sertous limitation of the results reported above Is the 
low response rate achieved in the SAES pilot test. If rhe response rate 
had been at ^east over (and preferably In the 70% range) » and if the 
results of the nonrespondent bias check had been the same as reported 
here (i.e.* no significant differences between respondents and oonrespon* 
dents)« the caution expressed at this time would not be so einphatlc. 
Readers should be aware* however* that all results reported here 
(especially from the correlation analyses) are based on a sflwil return 
rate. Although the results of the correlation analyses have been reported 
as preliminary evidence that there really are separate categories of 
skills necessary in evaluation activities* it should be remembered that 
these results are based on data from only 31% of the sample. 

Another limitation of the study Is the fact that the Role Description 
Questionnaire was not particularly appropriate. Respondents had trouble 
Identifying themselves according to the role descriptors. I isted* and it 
is unclear how that identification problem affected the way In which 
respondents provided information contained In the questionnaire. The 
SAES staff believes that a more effective means for obtaining the neces-* 
sary infonnation would be the method described In the ^'General Results" 
section of this report. 

As previously mentioned* It Is possible that the 2*500 names contained 
in the SAES staff's directory of educational evaluators was over-represented 
by college and university personnel. A more thorough search should be 
conducted before a full survey is undertaken to see if particular groups 
have been omitted from the list of names. State departments of education, 
large school districts* and professional education associations (besides 
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AERA) should be contacted and asked for help tn a>mpteting the directory. 

tt should also be mentioned that some of the persons to whom the 
questionnaire was sent wrote back to say they were not evaluators* tt 
is likely that sotne proportion of the nonrespondents considered them- 
selves not to be a member of the specified population and thus fatted 
to return the questic^nnaire. (The results of the nonrespondent bias 
check would confirm this possibility.) The SAES staff Is unclear about 
how this problem might be eliminated^ but it is definitely a matter of 
concern to anyone who might be contemplating further research in this 
field of investigation. 

The length of the SAES tns^trument was a definite drawback In the 
design of the pilot test. Item sampling procedures should have been 
'^Jmplemented at that time; had they been* the response rate would most 
likely have been higher* The decision was made by the SAES staff that 
such procedures would be used in the full survey* 

A final limitation of the study Is that there is no clear evidence 
of the validity of the self-^report data provided by respondents. With 
the SAES instrument as it Is now* all one has is the respondent's word 
that he or she Is highly competent In* e.g.* matrix algebra. There are 
no performance m**asures included in the SAES Instrument to test the 
validity of that statement* The inclusion of such performance measures 
would be a very useful addition to the SAES Instrument* 



Directions for Future Research 

The most obvious direction for future research in this area is the 

previously planned full survey using the SAES Instrument. Two SAES staff 

members have proposed such a survey and are currently attempting to obtain 
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funding for the study. The basic objectives of the study remain the same 
as when the Model Training Project was in existence^ but some revisions 
have been made as a result of the pilot test. Specif ical ly» the major 
objective of the study is as follows; 

1) To provide a description of the evaluation profession 

1.1) To generate a pool of items which represent the s.cope of 
competencies necessary for an evdiuator to perform in 
various roles 

1.2) To generate a pool of cognitive Items which measure the 
competencies in subobjective l.l 

t.3) To conduct a national survey of Individuals who fill ev^l*^^ 
uattve positions^ using the instruments designed in sub- 
objectives 1.1 and I .'2 

1.^) To provide a catalog of evaluator role profiles in terms 
of self-reported skiMs and objectively assessed cognitive 
beha or 

Two additional objectives relate to certain methodological considerations 

1) To test the use of overlapping matrix sampling (through item 
sampling techniques) in a factor analytic study 

2) To test the validity of self-report data by means of a comparison 
with objective cognitive measures 

Once funding for the full survey has been obtained^ a second pilot 
test would need to be undertaken. In this pilot test^ the item sampling 
procedures would be tested to make sure that the data obtained in this 
way from respondents would be appropriate for factor analysis. Assuming 
the results of that pilot test were favorable^ the full survey could 
then be conducted. _ 
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A word of caution should be injected at this point. Numerous persons 
have written to the HT(* asking for copies of the SAES Instrument. The 
SAES staff has been reluctant to release the instrument because of the 
meager reliability and validity data now available. The staff does not 
consider the instrument ready for use for diagnostic purposes for individ- 
uals at this time. The instrument ready to be used to d:>scribe groups 
of individuals* and that is the purpose of the full survey. 

In conclusion* it may be said that the Self Assessment of Evaluation 
Skills (SAES) Instrument was developed by Model Training Project staff 
for several purposes. One of the purposes was to empirically validate 
the Universe of Evaluation Competencies* a logical grouping of skills 
and knowledge required to perform the complete range of evaluation 
activities. After, numerous revisions* the SAES instrument was judged 
to be ready for use In a pilot test. That pilot test was conducted in 
late Fall, 1972. The results of the pilot test showed that with a few 
revisions (in items as well as in sample design)* the SAES instrument was 
ready for use in the full survey of educational evaluators which had been 
planned by the ^k^del Training Project. 

The staff of the Model Training Project at The Ohio State University 
agreed with the following assossinent of the state of training programs 
for educational research and research-related personnel (including eval- 
uators: 

One of the most serious impediments to efforts to 
plan or conduct training programs for research or research- 
related personnel in education is lack of knowledge 
about which particular competencies or skills are most 
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important in conducting research and research*reUted 
activities. ' 

The completion of the research begun by the SAES pilot test and described 
in this report would have been a valuable contribution to the knowledge 
base in the area of training in educational research and evaluation. 
Until that knowledge base is more complete^ It is unlikely tlmt viable 
training programs can be maintained. 



I 

Worthen^ B. R.* Anderson^ R. D.» & Byers^ H. L. A study of ' 
selected factors related to the training of researchers* developers, 
dtf^fusers^ and evaluators in education . Task Force on Training Educa- 
tional Research and Research*related Personnel^ American Educational 
Research Association^ November 1971. (Final report USOE Grant No. 
0EG-0-7I-06l7(520).) p. 15 
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Self Assessment of Evaluation Skills (SAES) Instrument 
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SELF ASSESSMENT Of EVALUATION SKILLS INSTRUMENT 
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t can ... ^ 
t. compare and contrast instructional research and evaluation. 

2. describe a recent trend of, manpower supply and demand In educational 
evaluation. 

3. design and administer evaluation studies that assess the effectiveness 
of projects. 

marshal political support for evaluation activities. 

5. prepare a 20-mlnute slldertape presentation on problems In evaluation. 

6. ' identify and analy2e t:.e prevalent government and foundation sources of 

financial support for research^ development^ and Instruction In educational 
evaluation. 

7. develop and h^lem|nt a systeiit for continually Informing system parsonnel 
about the work of an evaluation system. 

8. communicate findings from evaluation reports via television^ fadlo^ and 
newspapers in such a manner that the lay public can understand and critically 
consider the findings. 

9. describe and analyze several outstanding examples of educational evaluation 

systems and studies. 

10. design procedures which check the degree to which a curriculum package was 
implemented. 

11. critique an evaluation report for Us cost*ef fectlveness value. 

12. develop a schedule of reporting activities. 

13. design^ budget^ arrange for^ and support Internal audits of evaluation systems. 

Itf. plan specifically for the Involvement of system personnel in the development 
and implementation of the evaluation system. 

15. define roles that must be manned in efforts to advance the science and 
practice of educational evaluation. 

16. design and imple^nent evaluation studies that focus on needs^ problems^ and 
opportunities within the parent agency. 

17* set up a management system for evaluation programs. 

18* analyze the information requirements and reading levels of various audiences 
for evaluation reports. 

19* formulate specifications for a systems analysis. 
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20. coflvnunicate findings fro<n evaluation studies In meetings In such a ininner 
that profesifOnat educators and policy figures can use the Informttton to 
Influence their decisions and actions. 

2t. distinguish between research questions appropriate to the case study 
Rtethodology ahd'field study methodology. 

22, lead policy groups to adopt a sound set of evaluation policies. 

23. show that discriminant analysis Is a special case of canonical corre1a>Mon. 

2^, write both technical and popularized versions of an evaluation report, given 
the findings from an evaluation study. 

25. write an article that Is appropriate for subr^lsslon to gducatloral Projects 
Information Exchange . 

26. develop and implement an In-*service training program In evaluation- for 
persons at operational and managerial levels of the parent agency. 

27. communicate effectively In small group settings with policy groups such #s 
teacher associations and boards of education concerning the need for and 
nature of evaluation systems. 

28. apply specified criteria to alternatlve^program strategies In order to 
judge their relative merits. 

29. describe the use of judgmental data In evaluation and techniques by which 
judgments should be collected. 

30. develop funding proposals for programs designed to advance the profession 
of educational evaluation. 

31. set up review procedures for articles submitted to a Journal. 

32. develop and/or critique a handbook of polIct^s, organization, and procedures 
for an evaluation system. 

33. write an article that Is appropriate for submission to fh^.ipumalftf PHttcarlnnal 

Measurement , 

3^. list what audio-visual materials are available for the training of evatuators. 

35. design and conduct an evaluation of an evaluation system. 

36. while serving as an evaluator, develop and maintain rapport with the staff 
of a program being evaluated. 

37. project funding strategies for the evaluation system that Incorporate core 
internal <"pport, external support, and d fee structure for special evaluation 
services. _ 
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38. analyze a school En order to Identify evaluitton questions^ audtencet^ and 
information requirements. 

33. nonvlncingly address the lay public and professional educators conc«rnIng 
th*; ethical and educational merits of particular plans for collecting 
^v.jhicJtlon dato. 

^0, d«v<iiop,3 genurdl description^ for public consumption, of the 9oats^ services* 
^jfid overall philosophy of the evaluation system. 

^1. provide information vrfilch can be used to sell a curriculum package. 

^2. design and conduct a simulation study of alternative educational 

evaluation processes* 

k^i develop a taxonomy vrfitch can be used to categorize a body of printed matter 
in a specif ie d c onten t a rea,. 

discuss with policy boards of educational Institutions the policy Implications 
of evaluation. 

^5* formulate budgets for special evaluation studies. 

^6. develop and put to work an organizational structure for the evaluation system. 

^7. use audio-visual aids appropriately in making oral evaluation reports. 

^8. describe the problem of reliability and validity for criterion referenced tests. 

^9* describe the effect of different media on the message to be communicated. 

50. conceptualize a set of performance indicators that would form a sound basis 
theoretically and practically for a school district's systematic evaluative 
information system. 

51. conduct a comparative study among extant materials and a new currlcular 
developmenti 

52. describe the dynamics of small group behavior. 

53i communicate effectively with teachers and administrators in large group 
settings about the meaning of and need for policies to guide evaluation 
practices within an educational system. 

'5^. review and analyze the cost-effectiveness Implications of developing and 
implementing evaluation services. 

55. develop assessment strategies for the need for currlcular change, given a 
target population. 

56. articulate evaluation policies within the parent agency's policy framework* 
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57. present a 30'^mtnute address on new approaches to evaluation lo an audience 
of approximately 500. 

t ' Sb, present a case for the evaluation of competing Instructional strategies and 
suggest alternate methodologies which might be used. 

t 59. develop contracts with externa) agencies specifying the term of external 
3 eva-luation studies. 

^ 60. extrapolate from evaluation reports that assess competing program strategies 
to project cost and political Implications for the competing strategies. 

61. develop both 1ong<^range and short'^range projections of financial requirements 

\ for the proposed evaluation system. 

} 62. write an editorial policy statement for the operation of a refereed/ journal . 

r 63. write and enter into a sound performance contract with an outside agency. 

6^. design specifications for evaluation reports. 

65. design measuring devices which check for unintended outcomes. 

66. critique a test» having read a test manual for technical Information such 
i as norm group» reliability and validity. 

67. describe and analyze several major leadership programs of research^ develop- 
ment, and instruction in evaluation. 

68. establish organizational procedures for evaluation such that evaluation can 
have direct links to decision makers. 

69. discuss the assumptions underlying equal appearing Interval scaling techniques 
and techniques usiog Thurstone*s Laws of Categorical Judgments. 

70. organize and administer an editorial service tn relation to evaluation reports. 

7). make honest and systematic use of evaluation rn designing and carrying 
through a system of public Information. 

72. design evaluation feedback systems that define appropriate reporting settings^ 
content^ and medi':)» given an analysis of audiences to be served. 

73* determine and provide for office space» equipment and materials needed in an 
evaluation system. 

7^. lead policy groups to adopt a sound budget for system evaluation. 

75. provide stimulating leadership and direction to those who are serving in 
evaluation roles. 

.f 

76. provide parents with various types of evaluative data for their individual 
child. 

77. conduct survey research using a questionnaire, 

42 



39 

can * * * 

78. design, budget, arrange for, and support externa) audits of evaluation systems. 

79. negotiate a format and tiine schedule for evaluation reports to a cirent. 

80. develop goats and objectives for advancing the science and practice of 
educational evaluation. 

8). design an evaluation report to be presented by Video-tape. 

82. describe and analyze the major professional organizations that are active in 
the development of the educational evaluation profession. 

83. use reference sources such as Euros* Mental HeasuffflenU Yearbook or Tests In 
Print for the selection of a standardized Instrument. 

8^^. design and Implement a program for evaluating the operational levef staff/ 
such as teachers and school principal « of:an educational agency. 

85. — wopk-with^program„per-sonneJ_.to. ,ganerate^poJ.UieS- .and..guldeXlnM thajt wij I 

govern evaluation activities in the program. 

86. design and implement procedures for publishing and disseminating evaluation 
reports. 

87. write a formal critique for publication concerning the technical adequacy of 
a factor analytic study of evaluation roles. 

88. design charts — Such as histograms* trend line graphs* cross break tables* 
and pie graphs — to conmunicate evaluation findings both to professional 
educators and the lay public. 

89. assist in the revision of an evaluation activities schedule and In Its 
integration in the parent agency's master schedule. 

90. describe the National Assessment movement and performance contracting* drawing 
the implications of each for evaluation. 

91. use evaluation information to design* operational ize* and implement an 
agency accountability system* 

92* work with reference groups to anticipate decisions and associated Information 
requirements to be served by evaluation* 

93* design mechanisms to collect judgmental information concerning the worth of 
a curriculum package. 

94, design controlled research studies of small units In a curriculum. 

95. discuss the content o^^ varfouS articles in the AERA monograph series on 
currtculum evaluations 

96* develop a PERT network* 

97. explain the standard procedures used by most testing companies when normtng a 
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98. select^ organize^ and lead groups to develop criteria for judging competing 
program and project strategies. 

99- write an article that Is appropriate for submission to the R€vlw of Educational 
Research . 

100. describe the difference between Flanders* and Taba's Classrocxn Observation 
Schedules . 

101. design and administer evaluation studies that monitor the Implementation 
of projects. 

102. give a brief outline of the content of several texts which describe evaluation 
model s . 

103. describe the evaluation system to a group of evaluation theoreticians so that 
- they understand the basic conception of the system In relation to extant 

theoretical formulations in evaluation* 

10^. artTculate current problems in evaluation and where to find reading materials 
about them. 

105. teach teachers how to write behavioral objectives* 

106. describe the use of standardized tests for placement and diagnostic purposes* 

107. provide student outcome measures which can be Included as part of a curriculum 
package. 

108. recruit, orient^ and maintain communication with a pool of consultants who 
will perlodfcally assist In the work of the evaluation system. 

109. formulate specifications for an accountability system. 

110. describe basic concepts of analysis of variance^ e.g*^ design matrix^ random 
effect, interaction, confounding, and error. 

111. describe the difference between formative and sumntatlve evaluation in terms 
of rnforniation needs and Inferential base. 

112* use evaluation information to effect rational decisions about goals, strategies, 
management, and the recycling of activities* 

i 

113. Involve the lay public and professional educators In determining the questions 
to be addressed by an evaluation system. 

11^. '^cruit, select, and orient core staff for an evaluation system* 

115. communicate clearly and convincingly over television and radio about the 
nature and Importance of an evaluation system. 

116. critique an evaluation report for Its relevance to real decision problems* 
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118. 

119. 
120. 

121. 

122. 
123. 



construct simulation exercises which portray real evaluative problems. 



develop and disseminate clear and concise descriptions of evaluation systems 
that can be understood by the public. 

write forced-choice and free-choice test Items. 

organize groups of influential and professional personnel for performing tasks 
related to Improving the evaluation profession. 

define and maintain up^to'^date role descriptions and salary schedules for 
evaluation personnel. 

design and carry through an analysis of variance using a hand calculator* 

distinguish between revision cycles for Thurstonlan Attitude Scales and 
Likert Scales. 



12^. write an analysis of the relevance of economic theory to educational evaluation. 



125* design instruments which measure the cognitive behaviors delineated In a 
curriculum package. 

126. understand and judge the potential utility of outputs from projected data analyses. 

127* define a population and formulate sampling specifications for data collection 
purposes . 

. 

128. describe different usage of COBOL^ PORTIj^K» and assembly language to select 
the language best suited to a given evaluat Ion -problem. 

129. understand basic concepts of matrix algebra as they relate to statistics^ 
e.g., Inversion, transposition, rank, character isti c equat ion* conformabi I i ty» 
trace. 

130. anticipate decisions to be served by special evaluation studies. 

131. critique an evaluation design in terms of analysis and measurement assumptions. 

132. develop and present a major address on the state of the art In educational 
evaluation to audiences of educational researchers. 

133. describe the application of and the relationships existing among the major 
theoretical distributions of statistics, e.g., the normal, binomial, Polsson, 
chi-square, variance-ratio, and Cauchy distributions. 

13^. generate criteria with potential applicability for evaluating some generally 
defined developmental program such as "Individually Prescribed Instruction." 

^**>^'^ ^ ^ '* 

135. discuss various definitions of accountability and distinguish among account- 
ability, evaluation, and assessment. 

436. present a case for the evaluation of program goals. 



45 



c«n . * . ^2 

t37* identify a)te<^natlve analysts techniques and compare their appropriateness 

for a given set of data specifications which spell out des{gn» sampling^ and 
measurement assumptions* 

138* read with understanding most articles In PsVchometflka . 

139* design an experiment so as to allow analysis of Interactions between treatment 
and classification factors* 

140* design and Ifnplement a multiple regression type of study to assess the Influence 
of several variables^ Including provision of evaluation Information on decision- 
maki ng behavior* 

I'll* describe the basic structure and constructs of the FORTRAN language^ e*9*» 
type, dimension^ function^ equivalence^^ format* 

1^2* describe how validity coeffeclents are calculated and what they mean* 

1^3* compare and contrast the definition of true score In traditional measurement 
fheOry and the definition of tihlVerse score In general Izablilty theory* 

tlfif* perform an Item analysis and Interpret the results* 

write a formal critique for publication concerning the technical adequacy 
of a field experiment comparing alternative evaluation techniques* 

discuss the difference between orthogonal comparison and trend analysts* 

\U7* distinguish between a psychometric Inference and a statistical Inference* 

IJ48* project and define a number of research problems that might be pursued 
programmatical ly. 

\h9* develop and implement means for responding quickly and directly to requests 
for assistance In designing special evaluation studies* 

150* design an evaluation of extant currlcular packages In a specified content area* 

151. describe basic concepts Involved In parameter estimation* e*g*» blas» consistency^ 
likelihood ratio, etc. 

152. write operational objectives and specify attainment criteria associated with them* 

153. describe the functions of an evaluator In a targe curriculum development project* 
15^. design and develop a data bank* 

155. identify and select frm among the available measuring Instruments, given a set 
of information requirements. 

156* develop forms and procedures for managing the data processing operations of an 
evaluation system* 
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t57. discuss critically emergent developments In the field of educational measurement* 

158. Interpret test results given In standard score format such as grade equivalent, 
age equivalent, percentiles, deciles, or stanlnes. 

1S9* write a test blueprint. 

160. design procedures for assessing various media presentations of curriculum content. 

161. suggest techniques appropriate to measure degree of attainment for affective 
objectives. ^ 

162. write an analysis of the relevance of demographic analysis to educational evaluation 

163* design a sampling plan for field testing materials* 

^ 16W— -ijnderstand evaluation reports which Incorporate discriminant analysts, factor 

^ analysts, regression analysis, and .multivariate analysis of variance and covartance. 

165* organize and administers st^nclar^iTied^tesftlig pro 

166. design and tmptetnent a field experiment comparing alternative evaluation techniques, 
e.g.. Item sampling versus examinee sampling* 

167- analyze data from free response Interview data collection methods. 

168. design and conduct a historical study of the development of educational evaluation 
theory and practice covering a 10-year period. 

169* distinguish between situations where erosive unobtrusive measures are more 
appropriate than accretive measures. 

170. discuss the logic of statistical analysts, the major classes of questions that 
can be addressed by present modes of analysts, the classes of assumptions that 
can be accommodated by present statistical technology, and the emergent develop- 
ments In the field. 

171. apply appropriately the major nonparametrtc techniques, e.g*, Hann-^Uhltney, 
WItcoxon, Kruskat-Wat 1 Is, Friedman, Kolmogorov-Smt rnof f. 

172. identify, compare, and contrast the major approaches to measurement that 
have application In educational evaluation* 

173. formulate a researchable hypothesis appropriate to a tightly controlled laboratory 
study. 

17^* describe basic theoretical concepts relating to distrlbution^free statistics, e.g., 
robustness, asymptotic relative efficiency, stochastic Inequality, location and 
sea le. 

175. provide strategies to assess content for Inclusion In or exclusion from proposed 
curriculum packages. 



176* select a test that has been normed on an appropriate group. 

177. understand evaluation reports which Incorporate means, standard deviations* 
percentiles, grade^equlvalent norms, correlations, and analyses of variance. 

178. describe evaluation functions served by various offices within a university. 

179. compare the predictive validity of performance measures and simulation exercises. 

180. identify and define briefly major theories relating to learning and development, 
e.g., Hull, Plaget, Gagne, Skinner, Bruner. 

181. discuss the behavioral objectives controversy In evaluation enumerating the 
relationship between the application of each evaluation model and the model *s 
dependence on behavioral objectives. 

182. review major writings In the area of decision-making theory and Incorporate 
their main ideas Into a discussion of the utility of extant theoretical work 
In educational evaluation. 



183. systematically assess and judge the merits of an evaluation plan. 

18^. develop a monograph that Identifies, compares, and contrasts the major extant 
theoretical formulations in the field of educational evaluation. 

18^. design and carry through computer-based analysis of variance* multiple regression 
studies* and factor analysis using canned programs. 

186. describe and analyze the present state of development of the educational 
evaluation profession, as contrasted with those of educational research and 
development* educational ad ministratio n* and psychology. 

187. design and administer programs of research, development* Instruction, and 
service for the improvement of evaluation. 

188. describe the historical development of evaluation models and compare and contrast 
focal points of each. 

)89, conduct a literature review, write It up* and present It to a curriculum develop- 
ment group. Irrespective of substantive area. 

190* write affective objectives at various levels of the affective taxonomy. 

191. relate the major research strategies and substantive areas of philosophy and 
history to evaluation problems. 

192, design measuring instruments for affective objectives of a curriculum package. 

193- formulate and explain a set of criteria for judging theoretical work In educational 
evaludt ion. 

interpret standardized test results to parents and students. 
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195. present the main principles of Information theory and use them to critique 
the adequacy of present models of educational evaluation* 

196. state how reliability coefficients are calculated and what they mean* 
197* chair a curriculum evaluation connmlttee* 

198- compare and contrast the roles of an Internal and external evaluator In terms 
of focus. Information sources, and credibility* 

199* develop budget forms and procedures for an evaluation system* 

200. delineate evaluation authority and respons )bl I )ty within the agency's organizational 
structure. 

20t. organize and administer a data processing center related to meeting evaluation 
objectives. 



202. compare artd contrast traditional site visit methodology to advocate team 

" metho<iol09yr - — 

203. '1st and describe major Information processing concepts* e*g*» multiprogramming, 
random access* buffered I/O, Indirect and relative addressing^ and subroutlning* 

20't. design a study to evaluate the goals of a new currlcular development* 

20$. critique an experimental design In terms of Its Internal and external validity* 

206. compare and (Xtntrast the nature of educational evaluation with such related 
fields as research* development, planning* management* 

20/. relate the major research strategies and theories of sociology* pvlltlcal 
science and economics to evaluation problems* 

208. set up a framevK)rk within which program personnel can generate testable creative 
solution strategies for specified problems. 

209. interpret the computer printout for discriminant function analysis and HANOVA studies* 

2)0. provide specifications for and can control a management Information system that 
will maintain up-to-date information about program and project events and 
activities. I 

211, Select, organize* and lead groups of professionals to generate alternative 
program strategies and project designs* 

2)2. design a market research study for a new product. 

213. establish and implement quality control procedures for an information system. 

21't* relate the major research strategies and substantive areas of experimental and 
social psychorog^'^ evaluation problems. 
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218. 

219. 
220. 

221. 

222. 
223. 
22k. 

22S. 
226. 

227. 

228. 

229. 
230. 
231. 
232. 
233. 
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use the Delphi technique to assist groups to Identify and assign priorities 
to needs^ problems^ and opportunities* 

evaluate and/or design and conduct a case study of an o(>eratIonal evaluation 
sys tern. 

list and describe the characteristics and capabilities of major Infonnatlon 
storage media^ **9*> punched cards^ magnetic tape^ rum and disk* 

design and Implement (Including proper orientation for teachers) the conditions 
under which data are to be gathered* 

analyze the data produced In classroom observational study* 

suggest techniques appropriate to evaluate objectives at various levels of the 
cognitive taxonomy. 

describe evaluation theories^ models^ and practices In a wlie range of fields 
outside of education* 



design a study where path analysis Is the most appropriate analysis technique^ 

design procedures for assessing competing sequences of currlcular content* 

design an Information system according to which data are to be coded> stored> 
and- retr leved* 

specify^ operational Ize^ and ^pply criteria for evaluating evaluation systems* 

organize and administer a public information service In relation to evaluation 
i nformat Ion. 

defend the choice of an oblique rotation or an orthogonal rotation in a factor 
analysis study* 

state the basic principles of value theory and utility theory and compare and 
contrast the relevance of these fields for theory development In educational 
evaluat ion. 

write objectives at each level of the cognitive tax ^^ic^y* 
write a criterion referenced test. 

review historically the development of the legal bases for evaluation In education* 
conduct a training session for participant observers. 

design and Implement evaluation studies that identify dnd assess c(.>mpetlng 
solution strategies for specified problem areas* 



23^. describe and analyze user attitudes toward evaluation* 
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Commlttlon for m« Study of tti* EvaliMUon ProfMlon 



1 9n*m^rtit^*^ tn wftA^tiv^ 



November 10. 1972 



Ms* Anne P* Taylor 
P* 0* Box 603 

Correles* Ne^ Mexico 87^8 
Dear Ms* Taylor: 



T*m A t M Univtr^t^r 

imMtri* for Cduc«tb»A«i 0««4topiMni 

lltcHtf^A OMhim«f 

even O OutM 

J ThomM H*ttHk9* 
CinCC 

Wtc^irgv^ St«|* O«0<fW«Al ot education 

Malcolm M PtvM 
^0* v*r^nl* 



iWlphW. Ty(«f 



With support from the U.S* Office of EdocatJon* the 
Evaluacion Center of The Ohio State University, tn conjunction 
with the Cofnmission for the Study of the Evaluation Profession* 
rs collecting data to determine the range of competencies and 
interests in evaluation skills possessed by practicing cvalu*^ 
ators. and the relative Importance of these sicill* In job 
performance^ A full survey of the evaluatfon profession Is 
planned for early Spring; at present, we are concerned with 
refining the instruments to be used to collect these data. 
You ore receiving this letter because your name was drawn as 
part of a nationwide random sample of persons interested in 
educational evaluation. 

Several instruments are enclosed with this letter. One 
is the Self Assessment of Evaluation Skills (5AES). The Items 
tn rhi5 instrtinent comprise a sample of sictlls chosen from 
the Universe? of Evdluatinn Competencies* The Universe was 
developed through research at Ohio State combined with results 
frofT oreviocis svj^tas of evaluation ^kil) requirements* In 
the full survey cf the ev^aluation profession, ttie ta^k will be 
to determine how these skilU relate to diTferent evaluation 
roles. The data to be collected w*n be used to describe the 
skill configurations In various norm groups. The Role Descrip- 
tion Questionnaire Is included so we may gather information 
to be used to determine those norm groups* 

We would like you to complete these Instruments as if you 
were a participant in the full survey so that we may do prelim* 
tnary analyses to $e«» if the instruments are functioning as 
intended* In addition, the pt!ot sample has been divided into 
six subsamples; each subsampie is being asked to critique the 
technical adequacy of parts of the instruments. Fult direc- 
tions for performing these critiques appear later* All of * 
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your answers will be cccnpIetelY confidential; only group 
statistics will be reported. The ansv/er sheets and other 
materials are numbered to facilitate checking off rcspons^es 
as they are received. 

We realize that this is a big task v^e are asking you to 
perform* Wc are convinced, however, that this research is 
necessary to the grov/th of the fjetd of educational evatu^ition. 
Evaluation training programs need to train students so they 
will be qualified to accept "real world" positions in evalu- 
ation; but to date so Uttle research has been done in this 
area that training requirements relevant to real evaluation 
practice are unknov/n. Your participation in this study will 
add to the knowledge base in the fielc* o^* educational evalu* 
ation, and your cooperation will be greatly appreciated. 

Please return the SAES instrument, ftole Description 
Questionnaire, and all answer sheets in the pre*addfe£sed 
fetcrn envelooe which has been provided Tor your conveuience- 
If ^.'ou vioulcl like to have your name placed on the nailing list 
for the Interf acer . the newsletter of the Evaluation Center's 
Model Training Project, please fill in the yellovi card asking 
for your name and address and enclose it with the other 
materials in the return envelope. 

Thank you very much for your help in this endeavor* 



Sincerely, 




Enclosures 



53 



50 

Directions for Completing the 
Self Assessment of Evaluation Skills (SAES) 

The purpose of this study Is to coltect data on an evaluator*s competence and 
interest In specific areas of evaluation and related activities and hts perception 
of the relative Importance of each of these areas to his professional field. These 
norm group data r collected front a wide range of people presently engaged In evalu* 
ation activities^ will be used to valloate the Instrument so that It can 
be used to establish training programs for students in evaluation and to aid In the 
placement of students In various agencies. Complete data Is of the utmost Importance. 
Please answer each Item thoroughly and honestly. Your response will be held In 
strictest confidence; all data wltl be reported by group statistics only. 

Each ttemwill be rated according to three different scales: Competence, 
Interest, and Importance. Read each of the following scales carefully, then look 
ai the examples. These examples Illustrate how the three scales can be combined to 
form a single point of view concerning a particular Iton. After studying the scales 
and examples, mark each Item according to the three frames of reference by fllltng 
in the blank space under each scale with the number from that scale which b.est de* 
scribes your rating of that Item. Be sure to fill in all three scales for each Item. 
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Competence Scale 

^1% Ho CbrnPeterice ^ Respondent has no base from which to respond and would be com- 
pletely unable to perform this task. 

2, Minimum Competence - Respondent could perfofm this task minimally well only 

with extensive study. He does^ howevert know enough about 
the area to hire a knowledgeable consultant. 

3. Moderate Competence - Respondent could performthls task moderately wel I with 

some study and well with extensive study. 

High Compete nee - Respondent could perform this task well with minimal study and 
exceptionally well with extensive study. 

5. Superior Competence - Respondent could perform this task exceptionally well with 

tittle or no study. 



Interest Scale 

1. Negative Ipterest * Respondent finds this area repugnant. 

2. Mo Interest - Respondent has absolutely no Interest In this area. 

3. Mi nimal Interest - Respondent finds the area somewhat Interesting but can generally 

find more desirable activities to engage In. 

A. Moderate Interest - Respondent has genuine Interest In this area but there are 

professional activities he finds ix>re desirable. 

5. High Interest - There are few professional activities that the respondent would 

rather engage in. 

Importance Scale 

1. No Importance - Respondent almost never performs this task and considers It 

irrelevant to his professional performance. 

2. M i nima 1 Importance - Respondent occasionally performs this task but considers It ir- 

relevant to his professional performance. 

3. Moderate Importance - Respondent occasionally performs this task and considers It 

relevant to his professional performance. 

H ? qh importance - Regardless of the frequency of performance of this task* re- 
spondent considers It relevant to his professional performance. 

5* Crucial - Ability to perform this task* regardless of its frequency^Is vital to 
the respondent's successful functioning in his profession. 
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Examples 

Please refer to the preceding scales: 

Item: Can provide for necessary equipment acquisition and maintenance. 

Competence Interest Importance 

W (2) (5) 

The respondent feels that he Is highly competent In this activity but he 
has no Interest in It, even though he thinks the area Is a crucial one to 
his evaluation unit. 

ttem: Can write a questionnaire. 

Competence Interest Importance 

(^) (5) (2) 

The respondent feels that he Is highly competent In this area and tie finds 
It highly Interesting, although he thinks It Is Irrelevant to his particular 
professional field. 

Item: Can establish criteria for evaluating an evaluation system. 

Competence Interest Importance 

(2) (k) ik) 

Even though the respondent feels he Is only mlnlmalty competent In this 
area* he has a genuine Interest In the area and feels it is of high impor-> 
tance. 

The results of each Individual instrument will be confidential. Only the com- 
bined norm group data will be disseminated. Pleases feel free to answer each ttem 
frankly* Please do not omft any of the Items. 

Thdnk you for your cooperation. 
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ANSWER SHEET FOR SElF ASSESSMEMT OF EVAlUAirON SKIILS (SAES) 



DtRECTtONSr each set of parentheses below, write Che number which corresponds best to your perceived degree of 
^ competence, rnterest^ and importance for each of the items in the Self Assessment of Evaluation Skills 
(SAES)- -Write one number for each of the three scales for each item- Shortened definitions for each 
point on the scales are given on each page of this answer sheet; for a more detailed description^ 
please refer to the ^'Directions for Coin^Ieting the Self Assessment of Evaluation Ski I Is*' which accom- 
pany these matertals- 



COMPETENCE 



INTEREST 



IMPORTANCE 



K No Competence - completely unable to 

perform task 
2* Minimu'"* Competence - minimal performance 

with extensive study; able to hire 

knowledgeable consultant 
3- Moderate Competence - moderate perfor** 

mance with minimal study; good 

performance with extensive study 
k. High Competence - good performance with 

minimal study; exceptional perfor* 

mance with extensive study 
5* Superior Competence - exceptional perfor* 

mance with little or no study 



5* 



Negative Interest - area is 
repugnant to respondent 

No Interest - no interest in 
area 

Minimal Interest - area Is some- 
what Interesting; other 
areas more desirable 

Moderate Interest * area is 
genuinely interest I ng; 
other areas more desirable 

High Interest * few areas more 
interesting to responjdent 



I- No Importance * task almost never per* 
fomed by respondent; considered 
irrelevant to professional per- 
fo nnance 

2* Minimal Importance - task performed 
occasionally; considered irrel- 
evant to professional performance 

3* Moderate Importance * task performed 
occasionally; considered relevant 
to professional performance 
High Importance - task considered 

relevant^ regardless of frequency 
of performance 

5^ Crucial Importance task considered 
vital, regardless of frequency 
of performance 



OMF 



57 



INT 



COMP INT 



IMP 



IMP 



7* 
8. 
9- 
10* 
II- 
12- 



COMP INT 



COMP 



IMP 



INT 



IMP 



13. 

16- 

17- 

18. 



COMP INT 



COMP INT 



IMP 



IMP 



COMP 
19. ( 



20* 
2K 
22* 
23. 



INT 



COMP INT 



IMP 



IMP 
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COMPETENCE 



INTEREST 



IMPORTANCE 



1. No Competence - completely unable to 1. 

perform task 

2. Minimum Competence ' minimal performance 2, 

with extensive study; able to hire 
knowledgeable consultant 3< 

3. Moderate Competence - moderate perfor- 

mance with minimal study; good 
performance with extiensive study If. 

If, High Competence - good performance with 
minimal study; exceptional perfor- 
mance with extensive study 5- 

5. Superior Competence - exceptional perfor- 
mance with little or no study 



Negative Interest - area is 
repugnant to respondent 

No Interest - no interest in 
area 

Minimal Interest - area is some- 
what interesting; other 
areas more desirable 

Moderate Interest - area is 
genuinely interesting; 
other areas more desirable 

High Interest - few areas more 
interesting to respondent 



1. No Importance - task almost never per- 

formed by respondent; considered 
irrelevant to professional per- 
' forinance 

2. Minlmai"lmportance - task performed 

occasionally; considered irrel- 
evant to professional performance 

3. Moderate Importance - task performed 

occasionally; considered relevant 

to professional performance 
If. High Importance - task considered 

relevant, regardless of frequency 

of performance 
5. Crucial Importance - task considered 

vital, regardless of frequency 

of performance 



COMP INT 



IMP 



COMP INT 



IMP 



25. 
26. 

27- 
28. 

29. 
30. 

31. 
32. 
33. 
3if. 
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COMP -INT 



IMP 



35. 
36. 

37. 
38. 
39. 
ifO. 
ifl. 



If 2. 



if3, 



ifif. ( 



COMP INT 



IMP 



if6. 

i>7. 
If 8. 

50. 

51. 

52. 

53. 
5i>. 



COMP INT 



IMP 



COMP 



COMP INT 



IMP 



55. 
56. 

57. 
58, 
59. 
60. 
61. 
62. 

63. 
6if. 



NT 



IMP 



COMP INT 



IMP 
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COMPETENCE 



rKTEREST 



IMPORTANCE 



K No Competence ** completely unable to I* 
perform task 

2. Minimum Competence - minimal performance 2. 

wtth extensive study; able to hire 
knowledgeable consultant 3- 

3. Moderate Competence * moderate perfor-* 

mance with minima? study; good 
performance with extensive study 4. 

^. High Competence - good performance wHh 
minimal study; exceptional perfor- 
mance with extensive Study 5. 

5. Superior Competence * exceptional perfor- 
mance with little or no study 



Negative Interest - area is 
repugnant lo respondent 

No Interest * no interest in 
area 

Minimal Interest - area is some- 
what Interesting; other 
areas more desirable 

Moderate Interest - area is 
genttfnely interesting; 
other areas more desirable 

High Interest - few areas more 
interesting to respondent 
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COMP INT 



MP 



t. No Importance - task almost never per- 
formed by respondent; considered 
irrelevant to professional per- 
formance 

2. Minimal Importance - task perfomved 

occasionally; Considered irrel** 
evant to professional performance 

3. Moderate Importance - task performed 

occasionally; considered relevant 
to professional performance 

4. High Importance - task considered 

relevant, regardless of frequency 
of performance 

5. Crucial Importance - task considered 

vital > regardless of frequency 
of performance 





COHP INT 
65. ( ) ( ) 


IMP 

{ ; 


COHP INT 
75. ( ) ( ) 


tHP 


COHP INT 
85. ( ) ( ) 


IHP 


95, 


COHP 

( ) - 


IMT 


IHP " 


66. ( ) ( ) 




76. ( ) ( ) 




86. ( ) ( ) 




96. 


( ) 






67. ( ) ( ) 




77. ( ) ( ) 




87. ( ) ( ) 




97. 








68. ( ) ( ) 




78. ( ) ( ) 




88. ( ) ( ) 




98. 








69. f ) ( ) 




79. ( ) ( ) 




89. ( ) ( ) 




99. 








70. ( ) ( ) 




80. ( ) ( ) 




90. ( ) ( ) 




100. 








71. ( ) ( ) 




81. ( ) ( ) 




91. ( ) ( ) 




10}. 








72. ( ) ( ) 




82. ( ) ( ) 




92. ( ) ( ) 




102. 








73. ( ) ( ) 




83. ( ) ( ) 




93. ( ) ( ) 




103. 








7k. ( ) ( ) 




8^. ( ) ( ) 




9^*. ( ) ( ) 




m. 









COHP INT 



IHP 



COHP INT 



IHP 



COHP 



NT 



IHP 
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COMPETENCE 



INTEREST 



IMPORTANCE 



1, No Competence - completely unable to 1, 

perform task 

2, Minimum Competence - minimal performance 2« 

with extensive study; able to hire 
'knowledgeable consultant 3* 

3* Moderate Competence - moderate perfor- 
mance with minimal study; good 
performance with extensive study 
High Competence * good performance with 
minimal study; exceptional perfor- 
mance with extensive study 5* 

5* Superior Competence - exceptional perfor- 
mance with little or no study 



Negative Interest - area is 

repugnant to respondent 
No Interest - no, interest in 

area 

Minimal Interest - area is some- 2* Min 

what interesting; other 

areas more desirable 
Modtirate Interest - area is 3* 

genuinely interesting; 

other areas more desirable 
High Interest - few areas more ^, 

interestfng to respondent 



No Importance - task almost never per* 
formed by respondent; considered 
irrelevant to professional per* 
formance 
mal Importance - task performed 
occasionally; considered irrel- 
evant to professional performance 
Moderate Importance - task performed 
occasionally; considered relevant 
to professional performance 
High Importance * task considered 

relevant* regardless of frequency 
of performance 
Crucial Importance - task considered 
vital* regardless of frequency 
of performance 



ERIC 



105. 
106. 
107. 

?08. 
109. 

no. 
in. 
112. 
113. 
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COHP 



COHP 



NT 



IMP 



NT 



IMP 



n?- 
n8. 
119. 

120. 

12U 
122. 
123. 
}2k. 



COHP INT 



IMP 



COHP INT 



IMP 



125, 
126, 

127. 
128. 
129. 
130, 

131. 
132. 
133, 
13**. 



COHP 



NT 



fHP 



COMP I NT 



HP 



135. 
136. 
137. 
138, 
139- 
\kO. 

]k2. 
I'f3- 
lifif. 



COHP INT 



I HP 



COHP INT 



I HP 
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COMPETENCE 



INTEREST 



IMP0(\TANCE 



K No Competence - completely unable to 
perform task 

2, Minimum Competence - minimal p&rfonnance 

with extensive study; able to hire 
knowledgeable consultant 

3. Moderate Competence - moderate perfor- 

mance with minimal study; good 
performance with extensive study 
High Competence - good performance with 
minimal study; exceptional perfor- 
mance with extensive study 
5. Superior Competence - exceptional perfor- 
mance with little or no study 



K Negative Interest - area is I . No 

repugnant to respondent 
2* No Interest - no Interest in 

area 

3^ Minimal Interest - area is some- 2. 

what interesting; other 

areas more desirable 
Moderate Interest - area ts 3^ 

genuine I y ! nterest ing; 

other areas more desirable 
5. High Interest - few areas more 

Interesting to respondent 



Importance - task almost never per- 
formed by respondent; considered 
irrelevant to professional per-^ 
f Ormance 

Minimal Importance - task performed 
occasionally; consldw^ed irrel- 
evant to professional performance 
Moderate Importance - task performed 
occasionally; considered relevant 
to professional performance 
High Importance * task consl<{ered 

relevant* regardless of frequency 
of performance 
Crucial Importance - task considered 
vital* regardless of frequency 
of performance 



m. 

150. 

151. 
152. 

153. 
\5k. 



COHP INT 



HP 



COMP IKT 



( 

IMP 



155* 
156. 
157. 
158. 
159. 
160. 
161. 
162. 
163. 



COHP INT 



I HP 



COMP 



165. 
166. 

I67. 
168. 
169. 
170. 
I7L 
172. 
173. 



COH 



MT 



IMP 



COHP 



NT 



NT 



HP 



175* 
176. 
177. 
178. 
179. 
180. 
181. 
182. 
183. 
\8k. 



COHP 



INT 



IMP 



MP 



COMP INT 



MP 



65 



66 



COMPETENCE 



INTEREST 



IMPORTANCE 



1. No Competence - conspJetety unable to K 

perform task 

2. Minimum Competence • minimal performance 2. 

with extensive study; able to hire 
knowledgeable consultant 3* 

3. Moderate Competence - moderate perfor- 

mance with minima? study; good 
performance with extensive study ^. 

k. High Competence - good performance with 
minimal study; exceptional perfor- 
mance with extensive study 5. 

S. Superior Competence - exceptional perfor- 
mance with little or no study 



Negative Interest - area is 
repugnant to respondent 

No Interest - no interest in 
area 

Minimal Interesn - area is some- 
what Interesting; other 
areas more desirable 

Moderate Interest - area is 
genuinely interesting; 
other areas more desirable 

High Interest - few areas more 
interesting to respondent 



1. No Importance - task almost never per- 

formed by respondent; considered 
Irrelevant to professional per- 
formance 

2. Minimal Importance - task performed 

occasionally; considered irrel- ' 
evant to professional performance 

3. Moderate Importance - task performed 

occasionally; considered relevant 
to professional performance 
k. High Importance - task considered 

relevant, regardless of frequency 
of performance 
Crucial Importance - task considered 
vital, regardless of frequency 
of perfonnance 



COMP INT 
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) 



COMP INT 



MP 



COMP 



IMf> 



195. 
196. 
197. 
198. 
199. 
200. 
201. 
202. 
203. 
20^f, 



fJT 



MP 



COMP INT 



IMP 



20$. 
206. 
207. 
208. 
209. 
210. 
211. 
212. 
213. 
2]k. 



COMP INT 



IMP 



COMP 



NT 



MP 



215. 
216. 

217. 
218. 
219. 
220. 
221. 
222. 
223. 
22i^. 



COMP 



NT 



(MP 



COMP INT 



CO 



IMP 
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COMPETENCE 



INTEREST 



IMPORTANCE 



U No Competence - completely unable to 
perform task 

2, Minimum Cofnpetence - miMtmal performance 

with extensive study; able to hfre 
-knowledgeable consultant 

3. Moderate Competence - moderate perfor- 

mance with minimal study; good 
performance with extensive study 
High Coiiipetance - good performance with 
minimal study; exceptional perfor** 
mance with extensive study 
5- Superior Competence - exceptional perfor- 
mance with little or no study 



L Negative Interest " ^rea is 
repugnant to respondent 

2. No Interest - no interest in 

area 

3. Minimal Interest - area is some* 

what Interesting; other 
areas more desirable 
Moderate Interest - area is 
genuinely Interesting; 
other areas more desirable 
5. High Interest - few areas more 
interesting to respondent 



K No Importance * task almost never per- 
formed by respondent; considered 
irrelevant to professional per- 
formance 

2. Minimal Importance - task performed 

occasional ly; considered 'ixrel- — 
evant to professional performance 

3' Moderate Importance - task performed 

occasionally; considered relevant 
to professional performance 
High Importance - task considered 

relevant, regardless of frequency 
of performance 

5. Crucial Importance - task considered 
vital, regardless of frequency 
of performance 





COMP 


INT 


(MP 




COMP 


INT 


IMP 


roMP 


INT 


IMP 




COMP 


INT 


IMP 


225. 


( ) 


( ) 


( ) 


228. 


( ) 


( ) 


( ) 


231. ( ) 


( ) 


( ) 


233. 


( ) 


( ) 


( ) 


226. 


( ) 


( ) 


( ) 


229. 


( ) 


( ) 


( ) 


232. ( ) 


( ) 


( ) 


23^. 


( ) 


( ) 


( ) 


















COMP 


INT 


IMP 




COMP 


INT 


IMP 


227. 


( ) 


( ) 


( ) 


230. 


( ) 


( ) 


( ) 


















COMP 


INT 


IMP 




COMP 


INT 


IMP 
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Role Description duest tonna t re 

The following items witt provide us with descriptive data concerning your pre- 
sent position and previous position* Please fill in each blank* A list of roles 
is attached to which you may refer* Feel free to indicate combinations of the roles 
which are described below or add new roles* If you use one of the rotes listed* 
please list the name of the role and any additional information specified in that 
descript ion. 

Present Pos f t fon 

Title 

Rote Descr i pt ion/Ccnments (see trst below) " 



Department or Unit 

Institution or Agency 

Length of time at present position as of Oct* I* 1972 

PeQree refiuired for position 

Certification required for position ^ 

Most Recent Previous Position 

Title 

Rote Description/Comments {see li^t be tow) 



Department or Unit 

Institution or Agency 

Length of time at previous position_ 

Degree required for position 

Certification required for position_ 
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Role Description Questtonnatre (Con t . ) 



Degree Held - please supply information for the two most recently obtained 

Most Recent Next Most Recent 

Degree Held 

College or University ^ 

Year Completed 

Major Field 

Specific Area 

Aae Sex M F 



^ Role Descriptions 

K Administrator of Research ^ majority of your time is spent directing research 
projects* 

2, Administrator of Evaluation * majority of your time is spent directing evaluation 
projects (reviewing designs* budgets* proposals). 

3. Other Academic Administrator - majority of time is spent administering in an 
area other than research or evaluation. 

Teacher of Research - most time spent teaching research related courses (design* 
statistics* measurement). Please specify course area. 

5* Teacher of Evaluation - most time spent teaching evaluation courses (evaluation 
theory* design). Please specify course area. 

6, Teacher of Development - majority of time is spent teaching development lelated 
courses (techniques of packaged courses* innovations in education). Please 
specify course area. 

7, Other University Instructor - majority of time is spent as an instructor in an 
area other than RDD and Please specify area. 

8, Elementary / Secondary Teacher - majority of time is spent as a teacher in 
elementary or secondary schools. 

3\ 50% Teacher / 5ifA Researcher - t!me is split equally between teaching and 
research. Please note fields tn which you teach and do research. 
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Role Descriptions (Cont , ) 

10. 50% Teacher / 50% Administrator - time is split equally between teaching and 
administrating* Please note field Jn which you teach, 

IK 50^/- f^esearcher / 50% Administrator - time is split equally between research 
and administration. Please note field in which you do research, 

12, 50^?f, Teacher / 50% Evaluator - time is split equally between teaching and 
evaluating. Please note field in which you teach* 

13. 5(f& Administrator / 50% Evaluator - ttine is split equally between adini nistrati ng 
and evaluating* 

1^* 50% Researcher / 50% Evaluator - time is split equally between research and 
evaluation* Please note field in which you do research* 

15* Product Developer ' majority of tiine spent developing educational products 
(packaged courses* texts)* 

Diffusion Specialist - majority of time is spent in diffusion efforts (report 
writing* and presenting)* 

17* Evaluation Specialist; Design - majority of time is spent developing designs 
for evaluation of educational efforts* 

18* Data Analyst - most of time is spent analyzing data collected by others* 

19. Statistician - most of time is spent consulting with projects on statistical 
quest ions* 

20. Counseling ' most of time is spent counseling students in school systems or 
universities* 

21. Director of Standardized Testing - most of time spent directing efforts of 
school system or university* in collecting standardized test data* 

22* Evaluation Specialist; Implementation - majority of time is spent in the 
implementation of an evaluation design (e*g* data collection* reporting)* 

23. Federal Project Evaluator - majority of time is spent !n the evaluation of a 
federaMy funded program or project* Please specify type of project* 

2^, Evaluation Consultant - majority of time is spent in some other endeavor but 
often consultant help provided to other agencies in evaluation* 

Other • please list and define 
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Directions and /Answer Sheet for CritiquinQ I terns 
8e« Mes being interested in your answers to the items on the three scales, 
Vie would oppreciate it very much if you would help us assess the technical adequacy 
of the SAES i nstrument. We would I i kfi you to pay speci f ic attention to 5O *' tems , 
item numbers I through 50. (Other members of the sample are critiquing the 
remaining l8^ Items.) Please rate the clarity and ease of understanding of each 
of the 50 items by marking the appropriate option in the questions below. If you 
have written comments about the items, we would be very interested in seeing them. 



1 > I fc>und this item 

J ' completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 



1 found this item 
J ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



2. I found this Item 

( ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 



I found this \ tem 
X ) difficult to answer 
{ ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



3- I found this item 

( ) completely understandable 
( ) moderately clear 
{ ) moderately ambiguous 
( ) completely ambiguous 

Comments 



I found this i tem 
( ) difficult to answer 
{ ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



^> t found this i tem 

7 ) completely understandable 

( ) moderately clear 
( ) moderately ambiguous 
( ) completely ambiguous 



> found th i s i tem 

( ) di f ficult to answer 

( ) moderately difficult 

{ ) moderately easy 

{ ) easy to answer 



Comments 



5. t found thTs Item , 
( ) completely understandable 
( ) moderately clear 
( ) moderately ambiguous 
( ) completely ambiguous 



I found this Item 

1 ) difficult CO answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer. 



6k 



Cooinents. 



6. > found this^ I tem 

) completely understandable 
{ ) moderately clear 
K ) moderately ambiguous 
( ) completely ambiguous 



i found this Item 

7 ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



Comments 



> found this item 

( ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 



I found this item 
( ) difficult to answer 
( ) moderately difficult 
( ) moderatel ; easy 
{ ) easy to answer 



Comments 



t found this Item 

) completely understandable 



) moderately clear 
) moderately ambiguous 
) completely ambi9UOt*s 



I found this Item 
( ) difficult to answar 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



Comments 



I found this item 

( ) completely understandable 

( ) tnoderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 



t fot 

rr 



jund this Item 
difficult to answer 
( ) moderately cMfficuIt 
( ) moderately easy 
( ) easy to answer 



Comments 



to. 



f ound this Item 



ot 

} completely understandable 
) moderately clear 
) moderately ambiguous 



f 

( 

( ) completely ambiguous 



I found. this item 

J ) difficult to answer 

( ) moderately dirricuit 

( ) moderately easy 

( ) easy to answer 



Comments 



I I* I found this iten 

7 ) ccw)ple:ely understandable 

( ) moderately clear 

{ ) moderat<?'y ambiouous 

) ^ :'npl*: / 5^ J ^uoLS 



I found this item . 
( ) difficult to ''inswe' 
( ) moderately <i1t"f>ci; 
( ) mod *^tel' 

( ; 



Convnen.cs 



12, ' I found this Item 

) completely understandable 



) moderately clear 
) moderately a«nbtguou$ 
) completely ambiguous 



Cofnments 



t found this item 
( ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
) easy to answer 
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13. 



u nd this Item 
completely understandable 



{ ) moderately clear 
{ ) moderately ambiguous 
( ) completely ambiguous 



I found this Item 

{ ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

{ ) easy to answer 



Comments. 



I found this Item 

( ) completely understandable 

( ) moderaLety clear 

( ) moderately ambiguous 

( ) completely ambiguous 



und this item 



di f f Icul t to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



Comments. 



found this Item 
) completely understandable 
) moderately clear 
} moderately ambiguous 
) completely ambiguous 



I found this Item 
( ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



Comments 



16, I found this Item 

1 ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ar.Diguous 



1 found this Item 

1 ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( } easy to answer 



Comments 



17* 1 found this Item 

, 1 ) completely understandabl e 
{ ) moderately clear 
( ) moderately ambiguous 

( ) completely ambiguous 



1 found this item 

J ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



Comments 



18. I found this t tem 

1^ ) completely understandable 

f ) moderatelv clear 

( ) moderately ambiguous 



1 fou^d ' ^is it-sm 

( ) difficult to answer 

( ) .TOderatfily di f f icu 1 1 

C ) rodeT^a^'etv easv 

* ■% , - . 



Comments 
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20. 



2t 



22- 



I fyund this 1 tern 



completely understandable 
( ) moderately clear 
( ) moderately ambiguous 
( ) completely ambiguous 



Comments. 



I found this ttem 

) completely understandable 
( ) moderately clear 
K ) moderately ambiguous 
( ) completely ambiguous 



Comments^ 



t found this lt»m 

( ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 



I found this ? t£tn 
rr completely understandable 
) moderately clear 
) moderately ambiguous 
) completely ambiguous 



Comments. 



I found this ttem 



) completely understandable 
) moderate) y clear 
) moderately ambiguous 
) completely ambiguous 



i found this item 
( ) difficult to an5wer 
( ) moderately difficult 
( ) moderately easy 
) easy to answer 
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I found this I'teff 

( ) difficult to answer 
( ) moderately diff :cult 
( ) moderately easy 
( ) easy to answer 



I found this ilem 
1 ) difficult to answer 
( } moderately difficult 
( ) moderately easy 
( ) easy to answer 



1 found this item 

( difficult to answer 

( )'moderately difficult 

( moderately easy 



( 



easy to answer 



I found this item 
( ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



Comments 



25- 



2^. I found this Item 

rr completely understandable 
( ) moderately clear 
( ) moderately ambiguous 
( ) completely ambiguous 

Comments 



I found this item 



) completely understandable 
) moderatelv clear 
) moderately ambiguous 



; ccmpleteiv arnbtguous 
Comments 



I found this t teffl 
1 ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



t fou-^d ' -^is it^m 

( ) difficult to answer 

( ) moderately difficult 

( ) Mderate! v easv 



7' 



I found this Item 

( ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 



Comments^ 



I found this Item 

0 ) completely understandable 

( ) moderately clear 

\ ) moderately ambiguous 

( ) completely ambiguous 

Comments 



I found this Item 

7 ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 



I found this Item 

( ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comment s 



I found this item 

( ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 



I found this Item 

7 ) completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 



I found t hfs Item 
7 ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
(. ) easy to answer 
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I found this Ityn 

( ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



I found this item 
1 ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( } easy to answer 



I found this item 
( ) difficult to answer 
( ) moderately difficult 

( •) moderately easy 
( ) easy to answer 



I found this Item 

( ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



1- found this item 
7 ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



Comments 



I found th!s ftern 

7 ) completely understandable 

( ) moderatelv clear 

( ) moderately ambiguous 

( ) ccmptetely anbtsuous 



{ ) difficult to arswer 
( ) moderat^^ly diff icttt 
( ) moderate! V easv 



Comments. 
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33, 



34. 



■Lfflund this Item 

7 ) completely understandable 

( ) moderately clear 

( ) cmderately ambtguous 

( ) completely ambiguous 

Comments 



I found this Item 

0 } completely understandable 

( ) moderately clear 

X ) moderately ambtguous 

( ) completely ambtguous 



nts 



I found this Item 
( ) difficult 
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to answer 
( ) moderately difficult 
( ) moderately easy 
) easy to answer 



I found this Iten 
{ ) difficult to answer 
( ) moderately diff icuU 
( ) moderately easy 
( ) easy to answer 



35- I found this Item 

( ) completely understandable 

( ) moderalely clear 

( ) moderately ambtguous 

( ) completely ambtguous 



I found this Item 
{ ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



nts_ 



36. 



37. 



38, 



I found this item 

( } completely understandabl e 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 



I found this I tern 

7 } completely understandable 

( ) moderately clear 

( ) moderately ambtguous 

( ') completely ambtguous 

Comment s 



I found this Item 

completely understandable 
( ) moderately clear 
( ) moderately ambiguous 
( ) completely ambiguous 



I found this Item 

1 ) difficult to answer 

( ) moderately difficult 

( ') moderately easy 

( I easy to answer 



I found this Item 
( } difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



( found this Item 
( } difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



nts 



39* I . found th> s Item 

7) completely understandable 
( ) moderately, clear 
( ) moderately ambtguous 
( } completely a^nbiguobs 



I fou'^d '-'Is it* ,m 



) difficult to answer 
) modtrat**ly difficult 



( 

( ) moderate! v easy 



Comments 
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I found thl< lt€ffl 

( ) cofnptetety understtndabte 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous^ 

Comments 



I found thts Item 



our— 

0 ) completely understandable 
( ) moderately clear 
\ ) moderately ambiguous 
( ) completely ambiguous 

Comments 



I found this item 

( ) completely underi 

( ) moderalely clear 

( ) moderately ambiguous 

( ) completely ambiguous 



standable 



Comments. 



I found this item 
( ) completely understandable 
) moderately clear 
) moderately ambiguous 
) con^letely aAtblguous 



Comments. 



1 foiind^thls Item 
( ) completely understandable 
( ) moderately cl^ar t 
( ) moderateiy ambiguous 
( ) completely ambiguous 

Comments 



I found this item 

( y completely understandable 

( ) moderately cte.ar 

( ) moderately ambiguous 

( ) completely ambiguous 



Comments 



-LfOMnd this item 

1 ) completely understandabi e 

( ) moderatelv. clear 

( ) moderately ambiguous 

( ) ccmpieteiv ambiguous 

Comments 



I found this Item 69 
( ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
(, ) easy to ens%ifer 



t found this Item 
( ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) eas^y to answer 



I found this Item 

( ) difficult to enswer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



I found this Item 
1 ) difficult to answer 
( ) moderately difficult 

( *) moderately easy 
( ) easy to answer 



I found this item 

( ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



I feund this Item 
J ) difficult to answer 
( ) moderately difficult 
( ) moderately easy 
( ) easy to answer 



I fou^d ^ "'is it-^ 
J ) difficult to arswer 
( ) moderately difficult 
( ) moderatelv easv 
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I found this Item 

{ ) completely understandable 

( ) moderately clear 

( ) lAoderately ambiguous 

( ) completely ambiguous 

Comments,,,,,,, 

I found this Item 

) completely understandable 
( ) moderately clear 
\ ) moderately ambiguous 
( ) completely ambiguous 

Comments 

I found this Item 

( ) completely understjindable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments 

I found this Item 

( } completely understandable 

( ) moderately clear 

( ) moderately ambiguous 

( ) completely ambiguous 

Comments - 



I found this Item 
7 ) difficult to answer 
( } modetately difficult 
( ) moderately easy 
) easy to answer 
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I found this Item 

{ } difficult to answer 

( ) moderately difficult 

( ) moderatel y easy 

( ) easy to answer 



I found this Item 

( ) difficult to answer 

( ) moderately difficult 

( ) moderately easy 

( ) easy to answer 



I found this Item 
( ) difficult to answer 
( ) moderately difficult 

( *) moderatel y easy 
( ) easy to answer 
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Directions and Answer Sheet 
for Critiquing Rple Description ftpestlonnalre 

Besides being interested in your answers to the lt«ms on the Role Description 
Questionnaire, we would appreciate It very tnuch If you would help us assess the 
technical adequacy of the questionnaire. It should be noted that this Is * pre* 
llminary version of the questionnaire; In the full survey* we pl^*^ to have check- 
lists for the answers to many of the questions* We would appreciate your cociments 
on the item below* 

t* What sections of the questionnaire would be most conducive to the checklist 
format (e.g.^ for *%egree Held*** a list of degrees would be provided; the re^ 
spondent would check the degree which he holds)? 



2« ts the Information requested In the '^Present Pbsltlon'* and ''Most Recent Previous 
Position** sections clear? Do you have suggestions for clarifying the Item If 
you feet it Is necessary? , , , _ 



3. Is the information requested in the **Oegree Held*' section clear? Do you have 
suggestions for clarifying the item if you feel It is necessary? 
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Are the role descriptions provided clear? Do you fee? they are sufficient? Are 
there evaluation roles which were omitted and which you feel should be listed? 



5» Did you have trouble identifying yourself according to the role descriptions? 
If you did, How might the problem be alleviated? 
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APPENDIX C 
Follow-up Correspondence 
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CommlMion for tlw Study of tho EvalutUon ProfOMlon 



Evaluation C«ftt«r / Coll«9aofEducition / Tn«Ot)ioStitoUniv«f«lty / Columbut,Ot)to 43210 



November 3G* )972 



> Dear Colleague: 



Recently you received a letter from us requesting your 
assistance in a pilot test of the Self Assessment of Evalu* 
atjon Skills instruinent. To date* we ha^e not received your 
completed questionnaire* Your response is most important 
In order to have an adequate critique of the iteirs In the 
instrument. 



In case you might have misplaced the previous mailing, 
we are enclosing copies of the, materials we sent to you 
eariler* You will find another return envelope enclosed 
for your convenience. Your cooperation in completing and 
returning the materials will be greatly appreciated. 

Amort G A»h&Uff> * » » 

T«i«9 A & M Ur»v«t?itf 

Wiry Arnt 

*Th» 0^*t> Stit* UWruty 
Th« Ohio St«ft« Un'wfvity 

G*A« V GlM» 

ImWftiof or Edvci1iOn«i R«*«ifCti 
LNtiv«ftily of Cotorido 

EgonQ QuM 
ir^itnm Untvtrtity 

CiACE 

Ur>iv«/tity ot iiilfio*! 

iimMN Jtcot» 

Ortcmr M Puts*ic School* 

RiChiroM Jmo*' 
Univtfvit/ of Sooth Pkmdi 

MicfWO«n SUtB D«Oirtm«nt ot uucAlim 
Wlltiiffl B MiCti««l 
Unrv«rtity Of fU>uth«fn C*lifOfni« 

MikOffn M P#ovui 

LNtiv*rtttv Ot CAI^fOf nil it B«f««(*y 

OtAtttf Srjin«6««m 
Evalu«t^ C«fwt«r 
Tn# OWo Stitv UPiv«*ty 
AllpttW Tyftf 
Sc»tflci R*»flin;t) A««x-ii(«« 

WllUiffl W*t>flwr O n 

D«tiii fntf*(»n<i»ot School Ofitfict r) f 



Sincerely, 




Daniel L< Stufflebeam 
Study Co-directors 
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Commission for the Study of the Evaluation Profession 

Evaluation C«ntef / Conegeof Education / Tti«Ohio$tttftUniv0rrlty / Cotumbut.Ot)lo 43210 

December 15. 1972 



Dear Cot league ■ 



This is a last ditch effort " there will be no further 
follow*-up harassment beyond this reminder. But please, if 
Mt IS at all possible* >>fould you complete and return the 
Self Assessment of Evaluation Skills (SAES) Instrument which 
we sent tp you in Noveniber. 

In case you might have misplaced the previous mailing* 
we are enclosing copies of the materials we sent to you 
earlier. You will find another return envelope enclosed for 
your convenience. Your cooperation will be most appreciated. 



Sincerely* 



Af riotd G Alhtiuf n 
: TnttM A A M Un^vi^rltrr 

Josi^tur* fa* EdifCAttonn D***»ODfn*ftr 

M*fv AoF>^ BuAdJi 
EvtiuaTi^n C«nr^r 
T*\f Ofi*o STJlft Onivf^Ai 

Th^ OhpO Staff* Urwfffiity 

rtMfl V r*M« 

i^lRCE 

Uoiyo'l.ty Of $*hirn f lOnOJ 
Wiltpjvn ^ M'^hJ^i 

Evatti«iPon C<nr#' 

Th^ OhpO Sf Jlt^ I fn>V«r^tty 

Sc"»^<:o Ri^^urt^ Aifoc^am 
Ojii^^ indrtDrtt^c^nr 3^ hoot Oitum 




Study Co-d I rectors 



8(> 



November 30, 1972 



Dear Co) league: 

Recently you received • letter from us requesting your 
assistance tn a pilot test of the Self Assassment of Evalu" 
ation Skills Instrument. To 4at«» m have not received your 
completed questionnaire. Vour response Is most tnportant 
In order to have an adequate critique of the Items In the 
Instruments So (^teaset If you have not already done so, re- 
turn the completed quest loiinat re la the envelope trfilch was 
provided for you. Your cooperattoo wlH be greatly appre- 
ciated. 

, ^ Sii»c«re1y» 

Mary Anne Bunda ^ Oanlel L. %tufl99b€m^ 

Stuity Co^rfTrectors ' 



Deceviber <972 

D^^r U)t te^sgue 

This is VI U7St ditch effort — there v*nt be ihi 
farther foUoPr-u^* h^rttssment beyond this re^nlnder. But 
ple.i^^e, xf It is wll possible, imild you complete And 
return tne Self Asbessment of EvoIu**tlon Skills {SAtS* 
instrument iwhicn we bent to you In Hovember. Your cuup* 
er^ilcn wilt be mt-^t appreciated. 



ftdr^Hone Buod^ Daniel 
^ Study to-cllrector;^ 
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APPEKDIX D 
Item Contamination Substudy 
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CommiMlon for the Study of the Evaluation Profetalon 



Evaluation Cer«t&r / Colt«o« of Eoucation / TheOhloStatoUnlvarslly / Columbui. Ohio 43210 



November 17, )972 



Mr. Anthony Goeree 

Center for Vocational Technical Education 
igOC Kenny Road 
Columbus, Ohio ^3210 

Dear Hr^ Goeree: 



1 A/'VOti; O A^nlHjrn 
I O'^'i^ V 'jiM> 

I M<ico*(nM P/ft*u> 
i D4niH Siuf i^O^BFn 



With support from th*» U.S. Office of Education, the 
P'/aluation Center of The Ohto State University, in conjunction 
with the Commission for the Study of the Evaluation Professia-i. 
Is collecting data to determine the ran^e of coTipetenCies and 
interests in evaluation skills possessed by pract ic ing eval u* 
ators^ and the relat >\ importance of these skills in iob 
performance. A full survey of the evaluation profession fs 
planned for early Spring; at present* we are concerned with 
refining the instruments to be used to coMect these data. 
You are receiving this letter because your name was drciwn as 
part of a nationwide random sample of persons interested in 
educa r. I ona 1 eva I ua t ; on . 

Two MiStruments are enclosed with this letter. One is 
SAES-S. a shortened version of the Self Assessment or Evalu- 
ation Skills. The items in the ful! SAES instrument comprise 
a sample of skills chosen frtyn the Universe of Evaluation 
Competencies. The Universe was developed th^^ough research at 
Ohio State combined with results from previous studres of 
evaluation skill requirements. In the full survey o^ the 
evaluation p^^ofession. the task will be to determine how these* 
skills relate to different evaluation roles. The data to be 
col lected wi 1 1 be used to describe the skrll configurations in 
various norm groups. The Role Description Questionnaire is 
included so we may gather information to be used to determine 
those norm groups. 

We would like you to comt^lete These instruments if voi; 
were a partrCrpant in the full survey so That we trav cJo 
preliminary <^nd1yses to see if the instruments ^'^e fMnctiOOina 
as intended. AM of your answers will bf^ coi'^piereiv con-i- 
dential; only croup statistics wifl oe '-eoort^^c. Tnf* an^.ver 
sheets and other ir»aceria!s are nui^bered to ^^c^ii:aTe cne^.^ini 
off responses as they are received. 
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We feel that this research is necessary to the growth 
of the field of educational evaluation. Evaluation training 
programs need to train studeni;s so they will be qualified to 
accept *Vea1 world*^ positions in evaluation^ but to date so 
little research has been <)one in this area that training 
requirements relevant to real evaluation practice are unknown. 
Your participation in this study will add to the knowledge 
base in the field of educational evaluation^ and your cooper- 
ation will be greatly appreciated. 

Please return the answer sheets In the pre-addressed 
return envelope which has been provided for your convenience. 
If you would like to have your nanie placed on rhe mailing list 
for the Interfacer i the newsletter of the Evaluation Center's 
Model Trai ning Project . please fill in the form asking for 
your name and address and enclose It with the other materials 
in the return envelope. 

Thank you very much for your help in this endeavor. 



Sinceii,ly t 




StL'dy Co-di rectors 



Enc iosures 
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SELF ASSESSMENT OF EVALUATION SKILLS 
(SAES-S) 
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K prfe^er^^ a case for the evaluation of competing instructional strategies 
and suggest alternate methodologies which might be used for the evaluation 
of these strategies. 

57' present a case for the evaluation of competing instructional strategies 

17' suggest alternate methodologies which might be used for the evaluation 
of compet i ng i nst rue t i ona \ st rateg i es. 

2. design and implement procedures for publishing and disseminating evaluation 
reports * 

62' design procedures for publishing and disseminating evaluation reports. 

23^ implement previouf>1y designed procedures for publishing and disseminati 
evaluation report:;. 

3. design and conduct an evaluation of an evaluation system. 
55* design an evaluation of an evaluation system. 

63* conduct an evaluation of an evaluation system given an appropriate 
des ign . 

4. design and implement evaluation studies that identify and assess competing 
solution strategies for specified problem areas. 

16. design evaluation studies that identlfv and assess competing solution 
strategies for specified problem areas. 

60> iTiplement, given an appropriate design, evaluation studies that 
identify and assess cn^f^pfsting solution strategies for specified 
prob lem areas . 

5' describe the use of judgmental data in evaluation and techniques by which 
judgments sho- d be collected. 

38. describe the use of judgmental data in evaluation. 

28- describe techniques by which judgments should collected. 

6. design, budget. <*rrange for, and support external audits of evaluation 
sys terns . 

1*3^ design external audits of evaluation systens. 
8. budget, arrange for. and support external audits of evaluation systems. 

7. develop and P^»t to work an organizational structure for the evaluation 

sys tern. 

25. Implenent a previously developed organirat iojial structure for the 
ev<)luat ioi syste*ii. 
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21. develop an organizational structure for the evaluation system. 



9- develop and implement a system for continually Informing parent agency 
personnel about the work of an evaluation unit. 

12. develop a system for continually Informing parent agency personnel 
about the work of an evaluation unit. 

3K implement a previously developed system for continually informing 
parent agency personnel about the work of an evaluation unit. 

to* design and develop a data bank. 

16. design a data bank. 

^9. develop a data bank, given an appropriate design. 

II. develop and implement an in-service training program Tn evaluation for 
persons at alt levels of the parent agency. 

20. implement an established in-service training program in vjvaluation for 
persons at all levels of the parent agency. 

29* develop an in-service training Program in evaluation for^ersons at 
all levels 9^,^ t,he^^_paren agency. 

13* design and implement evaluation studies that focus on needs, problems, and 
opportunities within the parent agency, 

39' design evaluation studies that focus on needs, problems, and oppor- 
tunities within the parent agency. 

3. rmptement designed evaluation studies that focus on needs, problems, 
and opportunities within the parent agency. 

15* design and implement a sound program of staff evaluation. 

' 22. design a sound program of staff evaluation. 

58. implenient, given an appropriate design,, a sound program of staff 
eva I ua t ion , 

16. design and implement a field experiment comparing alternative evaluation 
techniques, e.g.. item sampling versus examinee sampling^ 

5^. design a field experiment conparing alternative evaluation techniques. 
e,g.. item sampling versus examinee sampling. 

27. implement, given a design, a freld experiment comparing alternative 
eva I ua t ion techn i ques . e . g , . i tem samp I i ng versus exam t nee sampl i ng- 

17* design and adnunister evaluation studies that assess the effectiveness of 
Pro Ject s , 
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19. 



7* design evaluation studies that assess the effectiveness of projects. 

64. administer^ given an appropriate design, evaluation studies' that 
assess the effectiveness of projects* 

provide specifications for and control a management information system that 
will maintain up-to-date information about program and project events and 
act i VI t ies . 



5. provide specifications for a management information system that will 
maintain up-to-date information about program and project events and 
act i vi t i es. 

14. control^ given sp^ci f ica t ions , a management information S/Stem that 
will maintain up-to-date Infornation about program and project events 
and act i vi t i es . 

20. use t aluation information to design and implement an agency accountability 
system. 

36, use evaluation information to design an agency accountability system. 

ho. Implement, given an appropriate design^ an agency accountability 
system. 

22* de^'gn and "Imp lemen t quality control procedures for en information system* 

k\ , design quality control procedures for an information System* 

56. implement previously designed quality control procedures for an 
i n format ion system* 

23* organize and administer a data processing center related to meeting evalu- 
at ion object i ves . 

34. organize a data processing center related to meeting evaluation 
ob j ect i ves. 

52. administer a previously organized data processing center related to 
meeting evaluation objectives. 

24, organize and administer an editorial service In relation to evaluation 
reports . 

2- organize an editorial service in relation to evaluation reports. 

48. administer a previously organized editorial service in relation to 
eva luat I on reports . 

26. design and conduct a historical study of the development of educational 
evaluation theory and practice covering a tO-year period* 
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50* design a historical study of the devetopment of educational evalu- 
ation theory and practice covering a lO-year period. 

. conduct, given an appropriate design, a historical study of the 
development of educational evaluation theory and practice covering 
a 10-year period. 

27* specify, operational ize, and apply criteria for evahiatrng evaluation 
systems. 

^5* specify and operational ize criteria for evaluating evaluation systems. 

37* ^PPiy a set , of operational ized criteria to the evaluation of an 
evaluat ion system. 

28. organize and administer a standardized testing program. 

23' organize a standardized testing program. 

9* administer a previously organized standardized testing program. 

31, design and administer programs of research, deveiofwrent, instruction, and 
ser^'ice for the improvement of evaluation. 

35* administer programs of research, development, instruction, and service 
for the imfrovement of evaluation. 

kk, design programs of research, development, instruction, and service for 
the improvement of evaluation. 

32. design, budget, arrange for, and support internal audits of evaluation 
system'".. 

13* design internal audits of evaluation systems. 

6k budget, arrange for, and support internal audits of evaluation systems. 

33' develop and disseminate clear and concise descriptions of evaluation 
s*''tems that can be understood by the public. 

33. develop clear and concise descriptions of evciluation systems that can 
be understood by the prjblic. 

IB. disseminate descriptions of evaluation systems to the appropriate 
public through various means. 

3^, organize and administer a public information service In relation to evalu- 
ation informat ion . 

2k, organize <i public information service in relation to ev<s!uation 
J nformat ion . 
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53* administer a preV ious ly organized public infomiation service in 
relation to evaluation i n Format ion . 

35^ design and administer evaluation studies that tnonitor the impletnentation 
of projects. 

S. design evaluation studies that monitor the ipplen^^ntation oF projects. 

conduct, given an appropriate design, evaluation studies that 
monitor the implementation of projects. 

36> design and conduct a simulation study oF alternative educational evaluation 
processes. 

iO, design a simulation to teach alternative educational evaluation 
processes . 

51* conduct a simulation, given the materials, to teach alternative 
educational evaluation processes. 

(8 Orlg/inal Single-verb SAES Items) 

8. describe Che use oF standardized tests for placement and diagnostic 
purposes , 



12. provide stimulating leadership and direction to those wKo are serving in 
eval uation roles . 

1^- use audio*vrsual aids appropriately in making oral evaluation reports. 
l8. develop a schedule oF reporting activities. 

describe the dynamics of small group behavior. 
2S' design a sampling plan for field testing materials. 

29, critique an evaluation report for its relevance to real deciston problems. 

30. interpret test results 9iven in standard score format such as, grade 
equivalent, age equivalent, pe^^cent i 1 es , deciles, or stanines. 
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AHSUER 5}fEET'rim SCU ASSESSHElfT OT'EVAltlATtON SKtUS (SAfS) 



OlRECTIOft^: 



.}h each „afrt of psreniKt-^ti belwf, Mtfte tlM.ntivber lAlch correipoittH^cb ybUr pert«i^d dtgree of cf*!^c*AC^..fnteteit* *Tt6 th^^'tAoce 
(For tach of th« lt«fn« tn the Self Alie»si)«ot of Ev^lvstlcn Ski Ms (SACS)* Vr)tt on«^,niii4wr for ekh of the three sc^tei for tach ft«3i* 
SHorttn«d ^rtnltlonl for tACh point on th* icites trt^.glwn iMtow: for,fi 9ore dlttan«it;d*icf1^t I6h. ple^ie refer, to tite 'tkir^tioni for 
;Cac^pt«tfng the S«^ir Afiei«n«nt of Evftlvat ton^SkM Ut> i^tch iccofpfny tt^se *^ *' > 



COHPETCWCE 



tNTEIlCST 



U *i6 Cbfrtpetence~^^co?Rt^t«lir vo«ble to ptrform \, 
h vitrniPvrCCvpet?^ perfoitrAnCe wl th e^'tcn'^ 

. / ^ i)y^ ,»tudy^: <ble to hire l[h<iHlc49C«bU coov 

liQNlerete\CCApcYtoc« * vodtrate p«ffor«iiinc« With ^tntMl ^ 

^ itud/;- 9004 ptrfofii^^ iiv^ 
.Mgh.Ctt^peteAcev^ 9m p«rfi6fWMic« mlth mlhln^t »tii4/; 

ajic«^t ioner perfbhMftca with cxtenifv* itiidy 4^ 
'Svperlor Cwtp^tence <^'«xceptto<wt perforwine^^l th 
ltttle:6riH> ^ti^rfy 

^ 5 



Me^ettvt tnte'r«it H repvg*, 

n«nt -to respondent' 
Nb tntereit no interest In ere< 
Hlfitvil; Interest '-^^er^a li jonp* 

whit lnterestiik9:roth«r ftre*i 

More defttr«bte 
Hoderete Intemt * eree U gtmir 

Iriely .ffiterefttfng; other ereM 

«ore d^lr«bt^ ' '$ 
High iMerest * feW ere^f jMore 

Intereittn^.Mi respondent 



^^I^ORTJUtCE 

Ho t*iport*nce r-t*ik-etno^t ooyer p^rfur*.^^ by v^trcn-lent; 

- considered trrelevent to prof^storij>l ptrfornvnce 
Htnlnst tpfkortence ^ »sk:perfoifted occ^siohalty: consldtred 

trrelevvnt ti>>roresston#l perfortiAnce 
HtHterntellMponahce jTjfik p«rf6r<»ed occasionally: eonstdered 

^ ./reljrVontMO^profeiitonal perfonMMce 
HtfH- t>N^r't«OM^*.ta^ re9erdtc«s of 

^'fre^uency 

. CructikI Npparten^/r/Kteik'Conftdtred vital, re^ordless of 
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ANSyEft SHEET fOft SFtf ASSE5£MfNT Of EvntfAtlON SICIUS (SAES) 



'plA£UK>iS; lA eftcHitet of p»r«ntbe«e« beTow, write the A^i4i«r whfch corre«pond» to your ptrc^lved dt^rc* of c€Ppet«f)ce« lAterv^tj. «M itM,t t'»*s<:« 
for each of ttke ltc»« Ia th« Self A»e«MicAt.of EvvluAtton SltlTI« (SAES). Write one mv*6«r for e*th of the thfte ic»le« for t-^tli Itent. 
:h<;rtened'cleflAttlonf» for s^ch potAt oo the «cele« 'Sre ^tveo i»cUH$ for e mre detailed deftcriptloOf pk^se refer to the **blfrct iort« for 
CbpplettAg the Self A«t«tt{heAt of Cv«lti*tlDA Ski IT«** >#hJch ncioft^p^hy ttke»o Mterl«l«« 



5. 



XOmPEIEHCE 

Wo torpet^Vtr ^ corpletely uAibTe to^perfom teth f. 
;jiiAip<^ C'/'&et«ncc * «>lolMt perfont4nce with eitten* 

jiJ*fe/»jti<f«.ible.to hire ltnbHled3e«4>le coftwNm 2. 
AG^fite^Cbr^tefkce * MHterete perforiwctt wtti) MtAtiiot 3. 

Itydy* perforvkAoce with ektenttve^itud/ 
Hl^h CoHiMtcAce '^. 9b6d;perfbnMnce With «HlotMl ^«ti*iV; 

ejuctptlOAOl perfonHMe wtti) eitt«n«tve «tud/ ^. 
5u(wrlor-Ccn|wtWe * excepttM perfonMnce wtti) 

little bf^no «tud/ 

- 5- 



INTEREST 



Ik9»tive lAtere«t - efce It repu^ 

n*At ^'to retpond«ot 
Mo Ifktcreit * AO interest Ia eree t* 
lllot«iiei jAtereit: * eree H tome- 

whit Interest tf»9; other ereos 3, 
^^POfc- ^tr«i>le 
tt&tferete^Ateriit *,ef«» tt 9«nu* 4« 
^fnely f Atere«ttf»9; other ereos 
piofc de«lr«4>te, ^ 5* 
Kt9h lAtere«t * 'fei* oVm Ppre 
/ lAtere«ttf»9 to retpbmlpnt 



II^ORTAlitC 

Mo l*9ort*nce r teth eTHDst-tvcvcr p^rforrrl* byrretpMfko* ; 

cOAflftefcd Irreiftv^'t'.ip f*ft-lottr6ti?l"perrorMnce' 
Hloittiet ipportiM'*' Uilt^p^^ cOA«l<ter^ 

f/releyeftt;.tdypfofe»itb«»rp^ 
rtod«roto:j«iportMee ^ t«eh pcrfojrped ocei9liN>eltv;.eefttl<l^re'd 

r^TeVMt !n o prof e « i I OAe| :pe 
HIglk Ifipi^tOTu;.* t«^t^^4^^ f«9*rdlm of 

freq^fkcy of~'perfbr«h^ 
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.ANSWER SHEET rOR SELF'^SSESSHEKT OF EVALUATIOH SKILLS (SAES> 

DJREC In each set of P«rcnthese b«1oH, nrltc the nunber hh I ch Torres ponds to your ^rc«1ve<f deg'^ coopotence. Interest. M importAnce for 

eacFT of the Itttns In th« Self Ms«S3Ai«nt of Ev«1u»tton Skills (SAES)v Vrlte otte lUMiber for esch of th« thm se«1es for each 1tat». 
Shorttned deflhltfons for each point oh the sc«1es *n gFven b«1<M; for„« nore detailed <lescrrptron^ .pl«m refer to the ''Directions for 
Collet Fng th« Self Ass«ssncfit of Ev«1uatron SkM Is*' which accotptny these ii»»t«r1«ts. 



COhPCTEtiCE 



IKTCKEST 



IMPORTANCE 



L No,.Cor«4-^nce ? conpletely tm«b1e to perfom t««k 1- 
'Z.»nMtntKi(^.tonpetence - attnlmvl pcrfOnn«ACe nUh eMter>* 

s<^' sttHf/^.jble to hire kficwfetf9e«b1« consultant 2 
3. HEM5cr«t«'^Coitp«t«nce r noderet* pcrfodMnce with RFnlinel 3 
st--«y; 900d petfonMnce h1 th extensFva study 
High Con^z^ncc *' go66 perf6m«nce nfth »fnJ«»1^ stu^: 
~ ejr^tiorVi perfbfiMnce nUh extensive study 4 
$i StipeHer iloi^oteMe * exceptronel pftrfontkMkce nlth 
rUtle or no stuity 

$ 



Hegetlve Interest- are* It repug* \. 
nmt to respondent 

No Interest - 4W Interest In ere* 2* 

H1nr*i«r Interest- eree Is some- 
what Interesting; other erees 3. 
«ore deslrabla 

Hodereta Intai^est - eree Is gemt- ^4 
inely Interesting: other erees 
nore desirable $* 

High interaif * few ereet Mre 
Interesting to respondent 



Mo Iviportance - tesk^ataost never. perfomed by respondent; 

considered Irretayant to professional performance 
Hlnfmal leiptM'tanca ~ task parfortbed occasionally; considered 

Irrelevant to/P^***1<^t perfonianca 
Moderate leiportance * task paVfOKfPed occasional ly; considered 

i^elavant to/profaasloheKparfocnanca -> 

High liipbrtanca:^* task consldered/retcvant. regardless of 

f re^Lancy^ of . parfo riiance 
Crucial fAtportanca * task considered vital* regardtasi of 

frequency of parfiorwioca 
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Commission for the Study of the Evaluation Profession 



Evaluation Center / College of Education / TheOhioSlateUnlveraUy / Colun;bui,Ohk) 43210 



- Dear Col league: 

Recently you received a letter from us requesting your 
assistance tn a pilot test of the Self Assessment of Evatu- 
atton Skills IhstrUrnent. To date* We 'have hot received your 
completed questionnaire. Your response t$ most Important 
tn order to hav^ an adequate critique of the Items In the 
Instrimtent. 



In ca$e you might have misplaced the previous mailing* 
we are enclosing copies of the materials we sent to you 
earlier* You^wltt find another return envelope enclosed 
for your convenience. Your cooperation In completing and 
returning the materials wltt be greatty appreciated* 




December 7* 1972 



Sincerely* 





\ "-iMdtoi* tor Education! D*veiofJmert» 



: J;,Ttiomft»H«lting< 



^ UniveiVty o< Colortdo 





Daniel L* Stuffl€A>eam 
Study Co-directors 



.; ^Iln1veisi(y ot South F\OitiiSi 
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APPENDIX E 

Eight Logtcally-dertved Categories of SAES Items 
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Category 

1, Knowledge of innovation 
in evaluation 

2* Public relations 
3* Data processing 

4« Educational measurement 

5. Evaluation a^lmjnlstratlon 

*- 

6« Relating evaluation to 
relevant disciplines 

7* Communications 

8« Research design analysis 



Itetn Numbers 

I, 2, 9, 10, 15, 21,29,30,35, 58, 67,80, 82, 
90,91, 93,95, 102, 103*104, 109, m, 120, 
131, 132, 134, 135, 136,153; 178, 183, 184, 
187,188,193,198,202,204,208,211,215, 
222,225,226,231,233 

8,20,39,47,53,81,115,234 

128,141,154,156,185,201,203,210,213, 
217,224 

34,55,66,76,83,97, 100,105; !06,I07, 
n9,t42, 144/152, 155, 157, 158; 159., 16^ 
165,167,169,176* i77,l8M90, 192*194, 
l96,i97,220,229*230V232- 

3,4,6,7, M ,13; 14, 16; t7;i9.,22v26i27, 
28 , 32 , 36 , 37 ,38 * irti; 44* 45 ,46 /54, 56 ,59; 
60,61, 63,68, 73,74,75, 78VB4*85,89, 92, 
96,98, 101 ,108, 112,113, 114, 1 16, 121, 
130,149,199,200 

124, 162, 180, 182,186, 191., 195,206,207. 
2)4,221,228 

5,12,18,24,25,31,33,41,43,49,52,57, 
62,64,70,71,72,79,86,88,99,118 

23,42,43,50,51,65,69,77,87,94,110, 

117,122,123,125,126,127,129.133,137, 

138,139,140,143,145,146,147,148,150, 

151,160,163,164,166,163,170,171,172, 

173 ,174-, 1 7571797189 ;2C>^20972 12,-2 1 67 

218,219,223,227 
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APPENDIX F 

Item Means, Standard OevJatJons, £• Inter-scale Correlations 
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ITEM OATA FOR SAES ITEMS 
(Heanst Standard Oeviations* |nten*scate Correlations) 

In the table below* item data for the 23^ Items .n the SAES instru- 
ment (divided Into the eight logically derived categories) are presented. 
Means and standard deviations for the competence scale are given since 
most other analyses described In this report pertain to competence only. 
In the inter^scaje^correlatlons, r^j is the correlation between competence 
and interest* is the correlation between competence end Importance^ 
and is tKe correlation between interest and importance. 

1. KNOWLEOGE OF INNOVATION IN EVALUATION 



Competence Inter-scale Correlations 



tern 


Mean 


Standard 
Deviation 


- !:ci - 


Tern — - 


r^Im- 


1 




.76 


.07 


.13 


.62 


2 


3.01 


1.18 


M 


.52 


.68' 


9 


3.58 


.85 


^ .39 


M 


.70 


10 


3.70 


.92 


.52 


.6k 


.70 


15 


3.16 


1.08 


M 


.5h 


.6k 


_2^1 — . — 


— ^.62 


1 .01 


.^9 


.38 


.63 


29 


3.55 


.93 


.Ul 


M 


.68 


30 


3.09 


1.13 


.59 


M • 


.51 


35 


3.36 


1.06 




A7 


.55 


58 


3.51 


.87 




M 


.62 


67 


2.96 


.97 


M 


M 


.69 


80 


3.35 


1.01 




.57 


.57 


82 


2.96 


1.07 


.^9 


.38 


.66 


90 


3.36 


.90 


.6: 




.63 


91 


3.10 


.95 


.59 


.US 


.68 


93 


3.6i» 


.92 


.39 


.30 


.76 


95 




.91 


.5^ 


.^•0 


.63 


102 


3.31 


1.00 


.60 


M 


.63 


103 


2.87 


1.01 


.63 


.5k 


.62 


10^ 


3.51 


.91 


.53 


.ki 


.67 


109 


3.16 


.92 


.65 


.30 


.60 


111 


3.69 


1.05 


.58 


.38 


.60 


!20 


3.03 


1.06 


.63 


.63 


.7^ 


131 


3.it2 


1.10 


.70 


.50 


.73 
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I. KNOWLEDGE OF IKKOVATIOK IK EVALUATION (cont.) 





Competence 




Inter- 


scale Correlat 


ions 


1 tem 


Mean 


Standard 


C 1 


I 


^ ill) 






Deviation 




132 


2.62 


.96 


.65 


.57 


.60 




3.57 


.88 ■ 


.61 


.61 


.66 


135 


3.51 


.87 


.53 


.39 


.61 


136 


3.87 


.82 


.62 




.7^ 


153 


3.77 


• ^ 


M 


.36 


.55 


178 


3.12 


1.13 


.&* 


.66 


.75 


183 


3.68 


.85 


.59 


.51 


.73 


18^ 


2.79 


.95 


.50 


.^5 


.68 


187 


3.26 


.91 


.73 


.^5 


.56 


188 


2.89 


.89 


.59 


A7 


.6^ 


193 


2.73 


.05 


.66 


.53 


.7t* 


198 


3.51 


.98 




.^9 


.59 


202 


2.71 


1. 00 


.60 


.70 


.69 


201f 


3.60 


.78 


.^7 


AO 


.5^ 


208 


2.99 


.93 


.59 


AO 


.61 


. ™2j 1 


3.22 


. .95 


.66 


.50 


.63 


215 


2^6„. 


1.19 


:6o 


.51 


.7^ 


222 


2.18 


.96 


.55 


.36 


.51 


225 


3.10 




.63 


M 


.58 


226 


2.81 


• 1.03 


.57 


Ai 


.52 


231 


2.32 


1.08 -^i^ 


.60 ■ 


M 


.72 


233 


3.09 


.92 




AO 





M. PUBLIC RELATIONS 



Competence 



Inter-scale Correlations 



1 tem 


Hean 


Standard 
Deviation 








8 


3.38 


K05 


,62 


.21 


A5 


20 


3.96 


.79 


.5^ 


.23 


.32 


39 


3,^9 


.93 


.58 


.39 


A5 


^7 


3,7^ 




.33 


.30 


.5^ 


53 


3.55 


.80 


.60 


A7 


.50 


81 


2.77 


,96 




.22 


.5^ 


115 


3.09 


1.09 




M 


.52 


■23^ 


3.39 


,95 


.56 


.38 


.6k 
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111. QAtA PROCESSING 



Competence tnter-scale Correlations 



1 tern 


Mean 


Standard 
Deviation 


r . 

^ 1 


•"cm 




(28 


2.12 


.87 


.25 


.16 


.37 




2.27 


1.02 


.36 


.3^ 


.31 


15^ 


3.17 


t.69 


.60 


.50 


.55 


156 


2.95 


t.l6 


.65 


.^3 


.38 


I85 


3i03 


1.36 


.70 


.31 


.^5 


?6l 


2.32 


' 1.03 


.56 


.Ik 


.^0 


203 


2.16 


.88 




.35 


..^9 


2~io 


2.83 ■ 


.97 


.66 


.50 


• .60 


213 


• 2M 


.99 


.57 


.36 


.52 


2,17 


* 2 .60 


1.02 


.58 


.27 


.56 


22^ 


2.76 


.t.05 


.5^ 


.27 


.56 



IV. EDUCATIONAL MEASUREMENT 
Competence Inter-scale Correlations 



Item 


Mean 


Standard 
— Deviation 


rci 


•* cm 


••im 


3^ 


2.73 


1.06 


.35 


.3^ 


.55 


55 


3.^9 


1.00 


.60 


.^6 


.65 


66 


3.83 


.98 


.67 


M 


.58 


76 


3.8i» 


.89 


.38 


.35 


.31 


83 


^.03 


1.06 


.55 


.51 


.75 


97 


3.6i» 


1.13 . 


.^9 


.^3 


.71 


100 


2.99 


, 1.01 


.52 


.^2 


Jk 


105 


k.\Z 


.91 


'.35 


.^3 


.63 


1G6 


^.09 


.93 


.^2 


.36 


.55 


107 


3.8^» 


.78 


M 


.35 


.63 


119 


^.00 


.82 


.37 


.30 


.65 


1^2 


3.^5 


1.09 


.67 


.51 


.5^ 


ikk 


3.8it 


1.06 


.52 


.^3 


.^0 


152 


3.95 


.86 


M 


.31 


.63 


!55 


3.66 


.91 


.68 


.^0 


.5^ 


157 


2.79 


.89 


.62 


.^6 


.60 


158 




.86 


.53 


.28 


M 


159 


l.ll 


l.^l 


.72 


.70 


.75 


161 


IM 


1.02 


.61 


.56 


.70 


165 


3.88 


I.I^ 


.53 


.36 


.57 


167 


3.39 


.88 


.56 


.55 


.66 


169 


2.61 


1.16 


.76 


.51 


.77 
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;V. EDUCATIONAL MRASUREMKMT (cont.) 
Competence Inter^scale Correlations 



1 cem 


Mean 


Standard 
Devi ac Ion 


r . 

CI 


t 

cm 


im 


176 


U.05 


.9^ 


.57 


.38 


.52 


177 


U.39 


.83 


M 


.36 


M 


18! 


3.29 


.98 


.6U 


.5t* 


.59 


190 


3.U5 


.98 




Mr ' 


.63 


192 


3.27 


.9^ 


M 


.37 ' 


.65 


19^ 




.81 


M 


.25 


.^3 


196 


3.79 


• 1.08 


.52 


.32 


.50 


-197 


3.61 


.93 


.50 


.52 


.59 


220 


3.56 


1.02, 


.60 


.51 


.75 


229 


3.87 


1.00 


A7 


.^5 


.66 


23d 


3.75 


1.03 


.62 


.57 


.70 


232 


3.^3 


1.03 


.5t* 


.50 


.62 



V. EVALUATION ADMINISTRATION 
Competence Inter-scale Corrtelattons 



tern 


Mean 


Standard 
Deviatluti 


"■ci 


""cm 




3 


3.97 


.87 


.38 


.32 


.51 


k 


2.75 


1.11 


.^9 


.58 




6 


3.05 


1.01 


.33 


M 


.^9 


7 


3.22 


.99 


.63 


M 




n 


2.88 


1.07 


.3^ 


.28 


M 


13 


3.06 


1.2U 


.55 


.60 


.60 


\k 


3.66 


.96 


M 


M 


.71 


16 


3.5^ 


l.l^f 


.70 


.59 


.76 


17 


3.08 


1.17 


.67 


' .^9 


.66 


19 


2.65 


1.07 


.56 


M 


M 


22 


3.^3 


.9^ 


.51 


.50 


.59 


26 


3.^9 


1.05 


.51 


.35 


.^7 


27 


3.89 


.93 


.63 


.38 


.39 


28 


3.69 


.83 


M 


M 


.50 


32 


3.38 


.93 


M 


.35 


.60 


36 


U.30 


.78 


.53 


M 


.55 


37 


2.76 


. :5 


.65 


.59 


.71 


38 


3.65 


1.02 


.52 


.60 


.5^ 




3.58 


1.07 


.69 


.38 


.51 




3.53 


.97 


.57 


.57 


.51 


^5 


3.26 


1.09 


.^3 




Ml 


U6 


3.^9 


.88 


.52 


.53 


.53 


5h 


2.77 


.97 


.65 


.^7 


.65 
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REWTING EVALUATION TO RELEVANT DlSCtPLlNES (cont.) 



I tern 


Competence 
Mean 


Standard 
Deviation 


Inter- 


scale Correlations 
""cm ""im 


195 


2.k2 


.79 


.51 


.36 ■ .65 


206 


3.29 


1.07 


.62 


.59 .62 


207 


2.51 


.88 


M 


.38 .66 




2.65 


.9^ 


.5k 


.^7 .57 


221 


2.3k 


.95 


.58 


.51 .61 


228 


2.18 


.83 


.ek 


.55 .62 



I tern. 



Competence 
Mean 



Vll. COMMUNICATIONS 

Inter-scale Correlations 



Standard 
Deviation 



'CI 



•^cm 



im 



5 
12 

18 
2k 
25 
31 
33 
^1 
k3 
kS 
52 
57 
62 
(,k 
70 
71 
72 
79 
86 
88 
99 
118 



3.31 

k.07 

3.08 

3.60 

3.k2 

3.21 

3.10 

3.36 

3.22 

2.65 

3.17 

3.26 

2.65 

3.53 

2.73 

3.25 

3.00 

3.kk 

3.3k 

3.87 

3-38 

3.kk 



1.08 


.kl 


.39 


.61 


.79 


.35 


.29 


.69 


I.U 


.60 


.50 


.62 


.9^ 


.51 


.27 


.kS 


1.10 


.52 . 


.Vf 


.70 


1.18. 


.^9 


.^5 


.53 


i.03 


.ks- ' 


.37 


.66 


.99 


.68 


.68 


.63 


.96 


.33 


.31 


.k2 


.9^ 


.56 


.kS 


.58 


1.06 


.58 


.kk 


.57 


1.02 


.52 


.3k 


.62 


1. 16 


.56 


.53 


.58 


.95 


.56 


.59 


.66 


1.03 


.57 


.^9 


.70 


.71 


.50 


.^7 


.67 


.89 


.56 


.^7 


.67 


.85 


.k3 


.37 


.63 


.98 


.ek 


.kS 


.67 


1.03 


.53 


.31 


.55 


.97 


.^9 


.^3 


.55 


.Sk 


.51 


.ko 


.58 
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VIII. RESEARCH _DES1GM AND ANALVS IS 
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APPENDIX G 
Revised SAES Items 
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—Revised SAES Items 



)n general, a)) double verb items noted in the Item Contamination 



Study should be divided into two separate single verb items, as shown 
i n Appendix- Or 

In addition, it is recommended that the following it&ns be revised 
or dropped from the SAES instrument which appears in Appendix A. 



I tem 
II 

2i 
37 



-5 
50 

51 

58 

60 
67 
69 



Recommendat ion 

Revise to read '*] can critique an evaluation report t6 
determine the cost-effectiveness of the program (or project^ 
materials, etc.) bei ng evaluated .** 

Drop from SAES 

Revise to read "I can project funding requirements (including 
core internal support, external support, and a fee struc- 
ture for external evaluation services) for the evaluation 
system J* 

Revise to read can provide information which can be 
used to effectively disseminate a curriculum package.** 

Revise to read *M can formulate a budget for a specific 
evaluation study.** 

Revise to read 'M can conceptualize a set of performance 
indicators that would form the basis for a school district*s 
evaluation- information system,** 

Revise to read ^*| can conduct a comparative study among extant 
curricular materials and a new curriculum package.** 

Revise to read *it can present a case for the evaluation '^f 
competing instructional strategies and suggest alternate 
evaluation methodologies which might be used.'* 

Revise to read **l can project cost and political implications 
From evaluation reports that assess competing program 
strategies.** 

Revise to read *'l can describe and analyse several major 
current programs of research, development, and instruction 
in evaluation. 

Revise to read **l can discuss the assumptions underlying 
equal appearing interval scaling techniques and Thurstonian 
scaling techniques.** 
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Reconvnendation 



81 

92 
103 

111 

12M 
126 

13^ 

li^l 

!m3 

157 

159 

162 

167 
169 
170 
178 
I8l 
182 



Revise to read **1 can design a video-tape presentation of 
an evaluation report.** 

Drop from SAES 

Revise to read 'M can describe a specific evaluation system 
to a group of evaluation theoreticians, relating t'le 
system to extant theoretical Formulations in evalu^-^tlon.'* 

Revise to read "1 can describe the difference between 
4 formative and summative evaluation in terms of information 
needs J' 

Drop from SAES 

Revise to read '*l can understand and judge the utility of 
projected data analyses." 

Revise to read "I can generate criteria for evaluating a 
developmental Instructional program such as * 1 ndt vtdual 1 y 
Prescribed I nstruct ion' 

Revise to read "I can describe the basic structure of the 
FORTRAN computer lai.^.age.** 

Revise to read "I can compare and contrast the definition 
of true score (from classical measurement theory) with the 
definition of universe scor^? (in general izabi 1 ity theory)." 

Revise to read "1 can discuss current developments in the 
field of educational measurement J' 

Revise to read can develop a general design (or test 
blueprint) to guide me in constructing a test.'* 

Revise to read "I can analyze the lelevance of demographic 
data to educational evaluation." 

Revise to read *'l can analyze data from open-^ended Interviews*" 
Drop from SAES or remove "jargon" from item 
Rewrite as four separate Items 
Drop from SAES 
Drop from SAES 

Revise to read "I can Incorporate major theories In decision- 
making into a discussion of their utility in educational 
evaluation." 
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I tem Recommendation 

185 Revise to read *M-can carry through computer*bascd analysis 
of variance, multiple regression studies, and Factor analysis 
using canned programs.'* 

186 Revise to read *M can describe and analyze the present state 
of development of the educational evaluation profession,*' 

189 Revise to read 'M can conduct a literature review in a 

substantive area, write a report from that review^ and 
present the report to a curriculum development group.** 

I95 Revise to read **1 can relate the major principles of Ipfor?* 

mation theory to current models of educational evaluation.'^ 

202 Revise to read *'| can discuss advocate team methodology and 

contrast it with traditional site visit methodoIogyV**"'^ ^ 

208 Revise to read **l can set up a framework within which program 

personnel can genf?rate solution strategies for specified 
* problems which are both testable and creative^." 

2l8 Revise to read *'l can arrange for the conditions undo*" which 

data are to be gathered (including proper orientation for 
teachers) .*' 

221 Revise to read **l can describe several evaluation theories^ 

models, and practices from fields other than education.*' 

228 Drop from SAES or rewrite as two items 
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APPENDIX H 

A Partial Valid^t\>n of the New Conceptualization 
of Evaluation Ccmpetencies 

Mary Anne Buhda 
Western Michigan University 
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A PARTIAL VALIDATION OF THE NEW CO((CEPTUAL(ZATION OF EVALUATION COMPET£NClE!:^v 

Mary Anne Bunda 
Western Michigan University 

The definition of ^ profession in terms of the universe of behaviors 
which comprise the activity in that profession Is not new* as Tom Hastings 
has pointed out (In another paper presented in the same AERA session). There 
is, however* a unique aspect to the Universe of Evaluation Competencies 
developed In the Ohio State Model Training Project. The uniqueness Is that 
we believe all individuals in the field have some competence across the range 
of skills; differences in areas of expertise may 1^^ reflected In a profile 
of thoce skills. The te5t of this assumption was planned through the develop* 
ment and refinement of the Self Assessment of Evaluation Skills (SAES) 
instrument. 

The initial draft of the Instrument was written witn the desire to 
delineate the universe of behaviors by which the evaluation profession Is 
characterized at all levels of Involvement and expertise. The inlttdl draft 
was divided Into ten major categories* each of which is necessary to build 
evaluation as a profession in the view of tne authors of the instrument. Thes 
cattigories are: (0 Administrative Leadership In Evaluation* (2) Research 
in Evaluation^ (3) Development of Evaluation. Methodology* (^) Instruction in 
Evaluation, (5) Implementation in Evaluation^ (6) Development of Evaluation 
Systems, (7) Diffusion of Evaluation Developments, (8) Use of Evaluation In 
Fducational Leadership* (9) Use of Evaluation in Teaching* and (lO) Use of 



^VA paper presented at the annual meeting of the American Educational 
Research Association* New Orleans, Louisiana* February 28, 1973. 

v^^:The author expresses appreciation to all Model Training Project staff 
whose work is reported In this paper; In particular to Ms. Evelyn Br^ezinski^ 
who designed the pilot study and has had significant impact on the direction 
of the research. 
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Evaluation in Curriculum Development, Tne hope was that this Instrument 
could be used to study the menibers of the evaluation profession and thus 
aid those involved in training evaluators. 

Although the evaluation profession nas been investigated several times 
?n conjunction wltn other professional educational roles (e.g.» researchers)* 
it has never been the topic of an in-depth study itself. Tne SAES instru* 
ment then» was to be used in a national survey of the members of the 
evaluation profession. A national study of those engaged {n evaluation 
3'ctiviVles would yield data which compre the competencies of persons in one 
role of evaluation activities (such as director of an evaluation unit) with 
those of persons in another role (such as technical assistant to an evalua- 
tion project}* Tne instrument^ therefore^ was constructed to Include items 
from various content areas. It was noped tnat no one content area was 
explicated better tnan another. 

Tnree major areas required further study before the national survey 
could take place: 

(1) Tne content of the instrument (i.e.> the items) had to be 
considered val id 

(2) Tne scoring system or profiling plan had to be sound. 

(3) Tne response format had to be valid and reliable. 

The initial procedure was'content validation of the items in the 
instrument. Tnis procedure nad two complementary activities. The first 
activity involved a search of extant literature which dealt with skills 
possessed by evaluators. Tne literature was searched for confirmation of 
the skills included in the draft items and for additional skills which were 
documented as necessary for successful evaluative operation. Tne success 
of the instrument was assumed to be based upon its scope. Redundancy of 

120 



107 



skills or slight differences in the presentation of skills were not con- 
sidered important factors. Rather^ primary Importance was placed upon 
scope and completeness. 

The second activity in content validation involved requesting leaders 
in the field of evaluation to review the SAES instrument. These individuals 
were sent the instrument^ asked to critique each Item* use the scale thaC 
had been developed to report their own skilly and add Items concerned with 
skills which had been omitted from the initial list. In this way two sorts 
of data were collected. The scope of the items was validated and somewhat 
enlarged, while the preliminary scoring syst«^ and response format were 
tested. 

The preliminary scoring system followed the ten roles which were used 
to develop the instrument and which were described earlfer. After data 
were collected in terms of the initial ten roles, several shortcomings of the 
scoring system were discovered. Firsts overlap among the scales was not 
consistent. Some roles were more Jnter^dependent than others. Second* ihe 
response categories which were constructed were not an exhaustive set. Con* 
sequently, a decision was made to develop a new scoring system based upon 
the results of a factor analysis of the instrument* to be performed after 
the data from the national survey were collected. Several interim scoring 
systems vserc used and one of them wiH be presenter* later In this paper. 

Other major changes occurred in the instrument in its early develop- 
mental stages* Most significant among these changes was the development 
of several response scales. Initially* data were collected only in terms 
of the competence a person perceived himself as having in each of the skills. 
The definition of these response categories is in Table I. Initially, it 
was assumed that this scale was rol>ust enough to allow for data collection 
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from Incumbent evaltiators^ from students In an evaluation training program^ 
and from learning experiences. If the computerized guidance system whtch 
Dan Stufflebeam described (tn a paper presented in the same AERA session) 
were to be implemented^ all three types of data would be necessary. However^ 
it soom became apparent that the scale lacked specificity for Individuals 
and was completely inappropriate for gathering datc^ about learning experiences 
Besides^ information concerning the present competencies of ibcumbent evalua* 
tors might not necessarily give good data about the training needs of the 
field, and therefore should not be automatically Inputted into a computerized 
guidance system. Thus^ the incumbent evaluator form of the instrument was 
revised to include two scales in addition to competence. The scales used 
in the pilot test are presented in Table 2. The importance scale was 
introduced to help delineate areas of skill which were perceived by practicing 
evaluators as important to sped fic roles In the field. The Importance 
scale Is a combination of frequency of use and criticality. These two 
factors jointly were believed to determine the necessity of a certain skill 
in any job. However, because the scale was not unidlmensional » many respond* 
ents had trouble using It. In future studies with the Instrument, the 
importance scale will have to be rewritten. The Interest scale was developed 
because It perhaps would be useful to know thei patterns of interest held 
by members of various roles in evaluation. This form of the instrument was 
administered orally to students In an evaluation training program and to a 
small group of professional evaluators. ^ 

Only 3 few minor changes in the wording of items took place after this 
test. The instrument was considered ready to be piloted on a group of 
Incumbent evaluators from our mailing list. It was hoped that the mailing 
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list, it was hoped that the maMing list would comprise all ir ividuals 
involved in evaluation activities throughout the country. Finding a 
list of such individuals proved to be no small task. An initial pool of 
names to be used in the full survey was the mailing list. of members of 
Division H of AERA. Before the list arrived, stratified sampling was 
planned, the stratifying variable being jobs as indicated by mailing 
addresses. The list, however, turned out to have home addressed for k7% 
of the division members. This caused us to abandon any hope of stratifi- 
cation by job. The list also failed to include many people who were known 
evaluators. Two plausible reasons come to mind immediately. First, Division 
H is relatively new in the organization and members of long-standing in 
other divisions may have failed to affiliate with Division H. Second, 
since the title of the division is School Evaluatior| and Program Development 
individuals lo evaluation roles in labs, centers, or universities may have 
felt that Division H does not represent their interest. In order to have 
a mailing list which adequately represents all evaluation roles, therefore, 
additions were made to the division list. Several sources were used to 
enlarge the list; (1) the present mailing list for the Interfacter , a 
monthly newsletter published by the Ohio State University Model Training 
Project, which included evaluators from the Model Training Project consortiui 
agencies, (2) personal address file of several evaluation specialists, and 
(3) two lists of evaluation consultants compiled by Bob Stake. The Itst 
that we used when drawing the sample for the pilot test, then, was composed 
of approximately 2,500 names and addresses. 

Since the hope was to perform a factor analysis on the instrument after 
the full survey, a decision was made not to item sample in the pilot test 
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but rather to ask each individual to respond to each of the 23^ items on 
each of the three scales. We realized that the length of the task would 
depress our response rate* Something had to be done to encourage respond 
ents to fill out the instrument. Firsts we formed a ''Commission for the 
Study of the Evaluation Profession** to endorse our study. The Commission 
\*tas balanced with leaders In the field of evaluation by geographical area 
and by their present role* That Is, we Invited Commission membership 
from the public school sector and the university sector of the evaluation 
profession. Each Commission member was asked for a commitment of at least 
two days of his time; the two days of work devoted by the Commission members 
involved a critique (prior to the pilot test) of the survey plan and Instru- 
ments to be used in the survey and a critique of the reporting format for 
the full survey* Permission was requested to list each Commlssjon member's 
»name. on the cover letter stationery used in the pilot test. 

The second strategy for improving our response rate involved a promise 
to respondents. In the full survey we planned to return to each respondent 
his score profile along with a booklet of mean profiles for each evaluative 
role. Since the scoring system was to be developed through a factor analysis 
of the fuil survey data, profiles would not be ready after tne pilot test. 
Also, the data collected In the pilot might not be representative enough 
(because of a low response rate) to create norm profiles. Rather than use 
an a priori scoring system and send out score profiles without the appropriate 
norms to t\eip interpret Individual profiles, another sort of *'pay-off** was 
promised to respondents. The Ohio State University Mode! Training Project 
at that time was, publ Ishing a monthly newsletter called the Interfacer which 
included spotlights on various consortium agencies, units within the 
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Evaluation Center^ and students in the Model Training Project. Respondents 
who completed the pilot test forms were offered the opportunity to be 
placed on the mailing list to receive that newsletter. 

In addition to the SAES Instrument^ a role description questionnaire 
was also included in the pilot test packet. The role description question* 
naire was developed for use in the pilot test and* in ^ revised format^ In 
the full survey. This questionnaire asked for various sorts of demographic 
and role data to help ascertain in what rote group an Individual belonged 
and to describe the norm groups in terms of academic experience^ average 
length of time in position* etc, in addition* twenty*four educational roles 
were described and respondents were asked to place themselves within ^ny 
one of the rotes or to describe their present position. Hopes were to 
refine this questionnaire to a checklist for the full national survey. 

The decision not to Item sample led to another problem besides the 
possibility of a low response rate. Tne mailing cost for each instrument 
packet was fifty-six cents. Efforts were made to study the conditions under, 
which we would get maximum response rate for the smallest mailing costs. 
Two major areas of mailing costs (reply format and followup format) were 
studied in the pilot test. A pilot test sample of 252 persons was randomly 
drawn from our population — 120 persons with home addresses and 132 persons 
with business addresses. These subjects were randomly assigned to a six* 
cell matrix which represented three different reply formats (stamped return 
envelope, business reply envelope* and return envelope with no stamp) and 
two followup formats (postcard follow-up or follow-up letter plus question 
nai re) . 

Tne results of the pilot test included the following: 
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(}) Of the 252 instrument packets malledt 77 usable responsest 

8 were incomplete responses* 21 were blank or undel { verab te* and 
1^6 were nonrespondents. Thus only ^2% of the questionnaires 
which were mailed ever found their way back to the Ohio State 
office. A telephone follow-up of a 15% random sample of the 
nonrespondents gave some leads to the causes of our low response 
rate. Of the sample nonrespondents » approximately 10% had 
cnanged positions and the forms had not been forwarded to them; 
arproximately 20% said they were not evaluators and therefore 
did not complete the forms; approximately 20% indicated that 
they had completed or partially completed the instrument and 
returned it (however* we never received it). Forty piercent of 
the sampled nonrespondents said they they did not complete the 
instrument because it was too long. 

(2) The results of the tests of the manipulation of response and 
follow-up formats were; 

(a) no ^significant difference In the return rate between persons 
who had received the questionnaire at home and at the 
office. 

(b) no significant difference In rate of return between postcard 
follow-up and letter plus questionnaire fol1ow**up either at 
the home or the of f i^e^sTddfeVr: I 

(c) no significant difference in response rates of the return 
format of the home address respondents. 

(d) a significant difference did occur in the response rates of 
the return formats at the office addresses. After one 
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follow^upt the business reply return rate was signt f tcantty 
higher than the no stamp return envelope at the :0S level. 
After the second fotlow-up notice ^he return rate ettcited 
by business reply envelopes was significantly higher than 
the no stamp returns at the .0? level and the stamped 
enve lope .return was greater than the no stamp return at the 
.025 level. 

Qn the basis of these statistics^ decisions were made which would 
cut mailing costs during the full survey (e.g*> business reply 
envelopes were chosen as most economical). 
(3) The roles into which our respondents were/classi fied are displayed 
in Table 3. As you can see> the majority of the respondents class- 
ified themselves into roles other than evaluation. The *'other 
administrator** group includes deans> school superintendents^ etc. 
The **other university professors** includes professors of» for 
example, educational administration^ curriculum^ and sociology. 
The group labeled **other** includes program and product developers, 
psychologists, secondary school teachers, and institutional 
researchers. 

It was hoped that the results of the pilot test could also include some 
typical profiles of several of the role categories. However, because the 
roles were so varied, any one role had very few individuals in it. There- 
fore, profiles drawn on the basis of any one group would be midleading. 
However, a sample profile of skills is presented In Table k for illustrative 
purposes. The two groups represented are administrators of evaluation or 
research and researchers or statisticians. The profile scoring system was 
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developed by sorting the SAES skills Into a priori categories. This sample 
proTile is presented to illustrate what thef results of the national survey 
might look like. Examples of the typ^ of Items tn each of the categories 
are listed In Table 5* The sample profiles are drawn for the competence 
scale only. It Is meant to be Illustrative of the differentiation of roles 
On an evaluation team In terms of competence 

Further analyses of the pilot test data are being completed at The 
Ohio State University. Included In these analyses Is a study of item 
contamination. As you probably noticed from the sample Items* some of the 
SAES items contain two verbs* such as 'Uo design and Implement". To deteilrithe 
if responses to double verb items differed from responses to items which con" 
tain 3 single verb* a small sub^study was conducted. Each of the 28 Items 
in SAES which contain a double verb was divided into two single verb Items* 
and three experimental instruments were created: 28 double verb Items* 56 
single verb items* and 28 single verb Items which are a random half of the 
total 56 single verb Items. Tne complete analysis of these data has not 
been completed* but will appear in the final report of the Ohio State Model 
Training Project. 
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ORIGINAL SCALE 



No Competency 

Respondent has no training base from which to work and would 
be completely unable to perform thts task. 

Htnlmom COfliPetencv 

Respondent has had some training and experience In this area* 
but does ^ot feel confident enough to perform the task without 
^further' training or complete supervision by someone knowledge- 
able In this area. — — ' 

Moderate Competency 

Respondent has had fairly thorough training and experience tn 
this area and could perform the task moderately welt* but his 
skills are not up to date and many other people in the field 
have much more expertise in this area. 

High Competency 

Respondent has a good background tn this ^rea and is com* 
pletely comfortable in performing this task. Some people in 
the field however* still possess more expertise In this area. 

Superior Competency 

Respondent is completely confident in performing this task and 

has done creative work .In this area. (S)He is one of the most 

skillful persons In the profession In this area. 
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SCALES USED IN PILOT TEST ' 
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Competence Scale 

I. No Competence - Respondent has no base from which to respond and would be 

completely unable to perform this task* or does not understand 
the terminology. 

2' Minimum Competence - Respondent could perform this task minimally well only 

with extensive study. He does* howevert know enough 
about the area to hire a knowledgeable consultant. 

3. Moderate Competence - Respondent could perfoim this task moderately well 

,wlth minimal study and well with extensive study. 

High Competence - Respondent could perform this task well with minimal study 
— and^exc^tionaUy well" with extensive study. 

5. Superior Competence - Respondent could perform this task exceptionally well 

without study. 



Interest Scale 

1. Negative Interest - Respondent finds this area repugnant. 

2. No Interest - Respondent has absolutely no interest In this area- 

3. Htntmal Interest -^Respondent finds the area somewhat Interesting but can 

generally find more enjoyable activities to engage in. 

High Interest * Res^^ndent has genuine interest In this area but there 
are professional activities he finds more enjoyable. 

5. Superior Interest - There are few professional activities that the respondent 

would rather engage in. 



Importance Scale 

1. No Importance * Respondent almost never performs thfs task and considers it 

irrelevant to his professional performance. 

2. Minimal importance - Respondent occasionally performs this task but considers 

it frrelevant to his professional performance* 

3. Moderate Importance - Respondent occasionally performs this task and considers 

it relevant to his professional performance. 

High Importance - ResponderiC often performs this task and considers it 
relevant to his professional performance. 

5. Crucial Importance - Ability to perform this task regardless of its frequency 

Is vital to the respondent's successful functioning. 
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Knowledge of Innovation In Evaluation 
M - 



Public Relations 
M " 7 



Data Processing 

~M----II 



Educational Measurement 
M • 3A 



Evaluation Administration Skills 
M • 50 



M Relating Evaluation to Relevant Disciplines 
M • 12 



>l Conmunlcatlons 
M " 22 



II Research Design Analysis 
M • 51 




M • 15 — — ——Researcher 
M • 15 'Administrator 

M • number of Items (n 
category 
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to 



n 12 



13 other 



1 administrator: research 
Z administrator: evaluation 
other administrator 



k 

5 
6 



professor: research cotirses^ 
(e.g., design, statistics) 

professor: evaluation coursas 
(e.g., theory, design) 

professor: developmrit courses 
(e.g., media utillzatlph. 
Instructional ilesigh) 



7 other college professor 

8 specialist: evaluation design 

9 specialist: evaluation 
Implementation 

1 0 consu 1 tant : eva I uat I on 

1 1 researcher 

12 statlsttclan/data analyst 

«13 evaluator of Federal projects 
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Table 5 
ITEM EXAMPLES 



I. Knowledge of Innovation in Evaluation 



I. I can compare and contrast instructional research and evaluation. 

58. I can present a case for the evaluation of competing instructional 
strategies and suggest alternate methodologies which mtght be used. 

93* I can design mechanisms to collect Judgrtental information concerning 
the worth of a curriculum package. 

211. I can select^ Organize* and lead groups of professionals to generate 
alternative program strategies and project designs. 

II. Public Relations 

^7* I can use audio-visual aids appropriately tn making oral evaluation 
reports. 

8K I can design an evaluation report to be presented by video*tape. 
23^. I can describe and analyze user attitudes toward evaluation. 
Ml. Oata Processing 

15^. I can design and develop a data bank. 

156. I can develop forms and procecjures for managing the data processing 
operations of an evaluation system. 

r 

22^. I can design an information system according to which data are to 
be coded* stored* and retrieved. 

IV. Educational Measurement 

3^. I can list what audio*vIsuat mat>rials are available for the 
training of evaljators. 

119. t can write forced-choice and free*choIce test items. 

176. t can select a test that has been normed on an appropriate group. 

V. Evaluation Administration 

I can marshal political support for evaluation activities. 

96. I can develop a PERT network. 

^ . 
200. I can delineate evaluation authority and responsibility within the 
agency's organizational structure. 
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Vr. Relating Evaruation to Relevant Olsclprines 

\Zk* I can write an analysis of the relevance of economic theory to 
educational evaluation. ^ 

207. . I can relate the major research strategies and theories of sociology, 
political science and economics to evaluation problems. 

228. I can state the basic principles of value theory and utJItty 

theory and compare and contrast the relevance of these fields for 
theory development in educational evaluation. 

VII. Communications 

5. I can prepare a 20-minute slide-tape presentation on problems in 
evaluation. 

T 

70* I can organize and administer an editorial service in relation to 
evaluation reports. 

118. I can develop and disseminate clear and concise descriptions of 
evaluationsyst6ms that can be understood by the public. 

VIII. Research Design Analysts 

Sk* I can design controlled research studies of small units in a 
curriculum. 

189. I can conduct a literature review, write it up. and preSent it to 
a curriculum development group, irrespective of substantive area. 

227. I can defend the choice of an oblique rotation or an orthogonal 
rotation in a factor analysis study. 
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